cases: Versioning Data and Models (rewrite) by jorgeorpinel · Pull Request #1747 · treeverse/dvc.org

jorgeorpinel · 2020-09-01T05:50:56Z

Continuation of #1716 (comment)

Moves the use case location and replaces all the links, adds a redirect.
Includes same related changes to other docs as cases: Versioning Data and Model Files (review) #1716
Removes term "SCM" in favor of "version control" (and other phrases) — see https://trends.google.com/trends/explore?geo=US&q=Source%20code%20management,%2Fm%2F015_g6,%2Fm%2F0fj8g
Redefines "data artifact" glossary term (could be removed from the glossary actually, not sure).

UPDATE: Pending stuff

PR scope cases: Versioning Data and Models (rewrite) #1747 (review)
2-column section? cases: Versioning Data and Models (rewrite) #1747 (review)

explain why versioning large files is important/a thing per #1716 (comment)

shcheklein · 2020-09-02T00:21:45Z

It's good in a sense that we are starting it from scratch 👍

Regarding the intro- it still does not give any sense why would versioning data and models is important (reference to a problem), how DVC solves it and why DVC is the way to go (selling the philosophy).

per #1747 (comment)

is important, and how it looks with DVC per #1747 (comment)

per https://github.com/iterative/dvc.org/pull/1747/files#r515533725

dmpetrov

It is a good writen use case. A few suggestions are inline.

dmpetrov · 2020-11-23T02:47:43Z

+
+- **Tidy project**: Work with a natural file structure. No need for ad hoc
+  naming conventions like `data/20190922/labels_v7_final`.
+- **Reproducibility**: Restore any project version and find the corresponding


A bit too abstract. I'd suggest moving it down in the list.

OK, I moved it down for now.

But I thought this was a pretty key purpose of versioning in general. Maybe we can rewrite it in a less conceptual terms, but I'm not sure how.

How can we make it less abstract?

it's an important point to my mind (even more important than a Tidy project (Simple names now))

OK I put in on the top 😆 for now. But we should come up with a criteria do decide which ones go first... After we decide which benefits to even keep (too many rn probably).

@dmpetrov 's point stays - if it feels abstract, should we reconsider the description? make Reproducibility and something in bold?

This is back around the middle and the other benefits have been generalized more so maybe now they all have a consistent level of abstraction?

dmpetrov · 2020-11-23T02:51:59Z

+  e.g. [CML](https://cml.dev/)), specialized patterns like
+  [data registries](/doc/use-cases/data-registries), and other best practices
+  that improve productivity and scalability.
+- **Data security**: The project history (in Git) is immutable, which allows for


I'd separate this into two:

Data security - set up security in underlying storage like S3 or NFS....

Data compliance - auditing processes to change data and models using regular GitFlow and Pull Requests. Visibility on data and ML model modification using a regular Git - who modified a dataset, when it was happening, who approved the change.

I agree that compliance is a better term, thanks! Updated along the lines you suggested.

Data security: maybe we don't need this benefit? Especially if we decide not to cover storage in this doc (per #1747 (comment) above). I removed it for now.

Data compliance should be probably part of the Git flow? or Git flow can be replaced by it?

I generalized all benefits into 5 now, PTAL. But the problem with Git workflows is that it has too many sub-benefits and it would be a huge paragraph if we include everything. I think Data compliance is a high-level benefit of it's own anyway?

We're even missing data security but per the comments above it sounds like it would be better to list it in https://dvc.org/doc/use-cases/sharing-data-and-model-files (as it has more to do with remote storage).

see #1747 (review)

per #1747 (comment)

and update benefits

…icated"

jorgeorpinel

Another iteration of this is ready for review. Here are 2 known (minor) issues I see:

…agaph (bottom)

shcheklein · 2020-12-05T19:10:26Z

+  complicated paths like `data/20190922/labels_v7_final` or for constantly
+  editing these in source code.
+
+- **Efficient data management**: Use a familiar and cost-effective storage


A final ask in private to update this a bit

shcheklein

Cool 😎 stuff!

jeremydesroches · 2020-12-10T05:04:22Z

@jorgeorpinel would you still like me to review this for SEO-related updates, as you mentioned before?

jorgeorpinel · 2020-12-12T06:09:48Z

That would be useful @jeremydesroches but completely up to you. Also at this point, since the rewrite was merged recently, do you think we should wait more weeks to be able to compare search patterns vs the previous version?

I guess giving it a read to make sure the key terms you added in https://github.com/iterative/dvc.org/pull/1806/files#diff-34b5f307478d996f5f763b6e71d8e9689543d9ef99f85d36a505c8ef47e57fee are still there (if possible) would be ideal.

Merge branch 'master' into use-cases

39d4400

shcheklein temporarily deployed to dvc-landing-use-cases-v-nb9tpb September 1, 2020 05:51 Inactive

This comment has been minimized.

Sign in to view

jorgeorpinel force-pushed the use-cases-versng branch from b7378a9 to df7a5f6 Compare September 1, 2020 06:00

jorgeorpinel temporarily deployed to dvc-landing-use-cases-v-nb9tpb September 1, 2020 06:01 Inactive

cases: [WIP] befin rewriting Versioning:

87264eb

explain why versioning large files is important/a thing per #1716 (comment)

jorgeorpinel force-pushed the use-cases-versng branch from df7a5f6 to 87264eb Compare September 1, 2020 06:01

jorgeorpinel temporarily deployed to dvc-landing-use-cases-v-nb9tpb September 1, 2020 06:02 Inactive

jorgeorpinel temporarily deployed to dvc-landing-use-cases-v-nb9tpb September 2, 2020 00:10 Inactive

jorgeorpinel force-pushed the use-cases-versng branch from d400921 to 87264eb Compare September 2, 2020 00:10

jorgeorpinel mentioned this pull request Sep 2, 2020

cases: Versioning Data and Model Files (review) #1716

Merged

9 tasks

jorgeorpinel added a commit that referenced this pull request Sep 2, 2020

cases: give some sense of why versioning data and models is important

6421771

per #1747 (comment)

jorgeorpinel temporarily deployed to dvc-landing-use-cases-v-nb9tpb September 2, 2020 17:25 Inactive

This comment has been minimized.

Sign in to view

cases: give some sense of why versioning data and models is important

8bdae1d

per #1747 (comment)

jorgeorpinel force-pushed the use-cases-versng branch from 6421771 to 8bdae1d Compare September 2, 2020 17:31

jorgeorpinel temporarily deployed to dvc-landing-use-cases-v-nb9tpb September 2, 2020 17:31 Inactive

guide: why DVC is the way to Version data (sell philosophy)

1679c59

per #1747 (comment)

jorgeorpinel temporarily deployed to dvc-landing-use-cases-v-nb9tpb September 2, 2020 19:22 Inactive

Merge branch 'master' into use-cases-versng

a794600

jorgeorpinel temporarily deployed to dvc-landing-use-cases-v-nb9tpb September 2, 2020 23:44 Inactive

jorgeorpinel commented Sep 2, 2020

View reviewed changes

Comment thread content/docs/use-cases/versioning-data-and-model-files/index.md Outdated

cases: add example section explaining why data versionig is

ab35693

is important, and how it looks with DVC per #1747 (comment)

jorgeorpinel changed the title ~~[WIP] cases: Versioning Data and Model Files (rewrite)~~ cases: Versioning Data and Model Files (rewrite) Sep 3, 2020

jorgeorpinel temporarily deployed to dvc-landing-use-cases-v-nb9tpb September 3, 2020 00:45 Inactive

jorgeorpinel commented Sep 3, 2020

View reviewed changes

Comment thread content/docs/use-cases/versioning-data-and-model-files/index.md Outdated

cases: wrap up Versioning full draft

5ed0d75

jorgeorpinel requested a review from shcheklein September 3, 2020 00:56

jorgeorpinel temporarily deployed to dvc-landing-use-cases-v-nb9tpb September 3, 2020 00:56 Inactive

jorgeorpinel added 2 commits November 22, 2020 13:58

cases: make first Vrsg figure shorter

98b3135

cases: merge advanced scenarios with benefits list

87b8333

jorgeorpinel commented Nov 23, 2020

View reviewed changes

Comment thread content/docs/use-cases/versioning-data-and-model-files/index.md Outdated

cases: roll back changes to Data Regs

59fc3f9

per https://github.com/iterative/dvc.org/pull/1747/files#r515533725

dmpetrov reviewed Nov 23, 2020

View reviewed changes

jorgeorpinel added 15 commits November 23, 2020 01:59

cases: improvements per Dmitry's feedback...

7677d25

see #1747 (review)

cases: train_feats > features in figures for Vrsng

a6f3352

cases: rename Vrng Tutorial label in nav (use emoji)

9644a38

cases: explain simple file naming a bit more

0e47977

per #1747 (comment)

cases: Vrng copy edits

8bd9d96

cases: add efficient data mgmt benefit

1686639

per #1747 (comment)

cases: reorder Vrsg benefits list

4ab5ed4

per #1747 (comment)

cases: rewrite file naming and data mgmt benefits of Vrsg

1b48db2

cases: expand story to cover storage and data management

0fbf200

and update benefits

cases: generalized Vrsg benefits

c7617ef

Merge branch 'master' into use-cases-versng

aa62d9d

cases: separate data mgmt from versioning (through codification) in Vrsg

c14a271

Make note about other guides, refs, and tutorial (Vrng)

020fb3f

cases: emphasize Simplicity benefit of Vrng is the opposite of "compl…

1a3a58c

…icated"

cases: another rewrite of text and benefits

f6598bd

jorgeorpinel commented Nov 27, 2020

View reviewed changes

Comment thread content/docs/use-cases/versioning-data-and-model-files/index.md Outdated

Comment thread content/docs/use-cases/versioning-data-and-model-files/index.md

jorgeorpinel added 4 commits November 26, 2020 22:32

Merge branch 'master' into use-cases-versng

98d97c7

cases: copy edits to latest Vrng iteration, and append next steps par…

6a75449

…agaph (bottom)

Merge branch 'master' into use-cases-versng

d6697dc

cases: another iteration of Versioning use case

2ccd32e

shcheklein reviewed Dec 5, 2020

View reviewed changes

shcheklein approved these changes Dec 5, 2020

View reviewed changes

cases: clarify data mgmt is for data in Vrng benefits

0d093b8

Conversation

jorgeorpinel commented Sep 1, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment has been minimized.

shcheklein commented Sep 2, 2020

Uh oh!

This comment has been minimized.

Uh oh!

Uh oh!

Uh oh!

dmpetrov left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jorgeorpinel Nov 24, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jorgeorpinel left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shcheklein left a comment

Choose a reason for hiding this comment

Uh oh!

jeremydesroches commented Dec 10, 2020

Uh oh!

jorgeorpinel commented Dec 12, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

jorgeorpinel commented Sep 1, 2020 •

edited

Loading

jorgeorpinel Nov 24, 2020 •

edited

Loading