guide: Data Management

[iesahin]

UPDATE: #2856 (comment)

---

This is the plan for data management trail that focuses on:

• Adding data to DVC projects
  & Versioning data in DVC projects
• The cache (local)
• & Shared cache (external)
• Removing data from DVC projects
• + Creating remotes (link to config/remotes)
• & Sync with remotes
  See also guide: extract remote add/modify details from cmd ref. #2866
• Accessing public datasets and data registries (get, import)
• External data topics (See how: use DVC when data is stored in an external drive #563 (comment))
  • probably address guide: consolidate external data mgmt guides #520

Details

Adding data to DVC projects

• Initialize a DVC repository and use dvc add to add files.

• We'll assume MNIST data exist in a folder and will add it.

Versioning data in DVC projects

• Overwrite Fashion-MNIST data on top of MNIST and update the dataset.
• Go back and forth in Git history to get different datasets in the same folder.

Creating remotes

• Add a Google Drive folder as a remote.

• Make it default

Pushing to/pulling from remotes

• Push the cache to the remote we created
• Clone the repository to somewhere (e.g. ssh or local folder)
• Pull the cache

Accessing public datasets and registries

• Get the Fashion MNIST data from dataset-registry

Removing data from DVC projects

• Remove certain folders from workspace
• Delete the corresponding cache files

UPDATE: start with a reorg, see #2856 (comment) below (may be enough).

[dberenbaum]

👍

The only thing I can think of that you might want to add is import/update.

[iesahin]

Actually, import/update might not be necessary since this is GS. In fact, I think you could probably stop the GS doc after "Pushing to/pulling from remotes."

Let me try to write something with the above plan and we can see what's extra. I think "public repository access" may be a good selling point. I believe these documents are like "brochures" and some kind of advertisement. I'm just trying to touch as many points without much technicalities.
@dberenbaum

[iesahin]

I think it's better to discuss the scope of the update here @shcheklein @dberenbaum @jorgeorpinel

• I'll remove gc and remove from the document
• Should I keep example-get-started intact? I'd use an empty repository and dvc add a file downloaded with wget. If I remove dvc import and code related tracking, I think this approach requires no code to be downloaded. People will just dvc init and start tracking their data.
• The current titles include model keyword, should I keep it or not? Data and Model Management may be too broad but I think models are also data.

[shcheklein]

Should I keep example-get-started intact? I'd use an empty repository and dvc add a file downloaded with wget.

sounds good as a first step. I would focus specifically on restructuring the existing sections. A good first step can be as simple as moving all of them under the Data Management Trail and figuring out the redirects, names, intro part on the index page, etc

If I remove dvc import and code related tracking

could you clarify please?

model keyword, should I keep it or not?

Good question. Pros are that a bigger title could resonate with a bit more people. What are the downsides? I don't have a strong opinion on this. I would still try to keep in the text models as much as it makes sense to keep highlighting that models could be managed this way. There should be a feeling that it fits into GitOps for models (w/o data) scenario very well.

[iesahin]

I would focus specifically on restructuring the existing sections.

From my experience with the experiments doc, restructuring usually takes more time and effort than a rewrite. The flow of the current documents assumes the user will have a project. I don't want to assume this in the Data Management Trail, the user might be looking to track their music collection with Git. We don't need code, we don't need Python, we don't need anything except a bunch of files to tell Data Management features of DVC. (Even Git is optional I believe, I would like to have a "how to add your photos and send to your friend in a DVC repository" kind of project rather than "data science", etc.)

If I remove dvc import and code related tracking,

I don't think we need a project to download/clone from somewhere to tell the data management. The current docs begin by downloading code.zip. Data management should better start with data, and end with it.

I would still try to keep in the text models as much as it makes sense to keep highlighting that models could be managed this way.

I think we can have a separate "Model Management Trail" adding experiments and MLEM to the mix. Most of the features are here, we'll have links to sections in Data and Experiments Trails' sections. "Data and Model management" can also have a section about how models are actually like data files. However, from the pointer Jorge shared in the PR, I think it may be better to call this "Data Management" and write another "Model management" document specifically about creating models with experimentation and tracking them with DVC. We already have these features, it will just be another narrative.

[shcheklein]

To be more specific:

restructuring == move/rename existing sections. Don't change anything else. E.g. create a new section "Data Management Trail" that has inside it exactly same sections that we have now.

The flow of the current documents assumes the user will have a project. I don't want to assume this in the Data Management Trail

we can get to this later. I think experiments is a bigger priority still. Or even getting pipelines out of data.

the user might be looking to track their music collection with Git.

We can discuss this when we get there, but it's an open question. My personal take for now, we should stay close to ML examples in the regular data management trail. We can mention that non-ML specific scenarios could be covered.

Also, keep in mind - we can't cover everything in the Get started. We should prioritize the most critical scenario, the most critical path, the most common set of commands. This is get started (aka quick start), not UG.

The current docs begin by downloading code.zip. Data management should better start with data, and end with it.

Current docs begin with:

dvc init
git commit
dvc add data.xml

Very precise focus, everything else comes secondary.

Or maybe I misunderstood what documents do you mean?

Intention was to hide everything ... even code download happens in the expandable section in the third section or something.

I think we can have a separate "Model Management Trail" adding experiments and MLEM to the mix.

may be, I would come to that after experiments, after the basic data management restructuring and when we have MLEM a bit more complete.

[jorgeorpinel]

re title I'd just avoid the phrase "model management" since it has a specific meaning (not what this is about) but if you want to include "model" maybe use "data and model file management". I don't think we need to include "model" in the title but "model file(s)" can be included in the content.

[shcheklein]

since it has a specific meaning

I'm not sure we have it written somewhere? :) what kind of meaning do you have? what is so different between model management and data management?

(I can see for example that if we include model management we could keep some parts about metrics for example - which is also fine)

[jorgeorpinel]

Like, related to the ML model lifecycle? I mentioned this in the PR (https://www.dominodatalab.com/solutions/model-management/) and it wasn't contested so I assumed I was correct but you guys are the experts! If it doesn't have a special meaning then it doesn't matter. But if it does users and search engines could get confused.