Skip to content

dvc: add Repo(skip_checks) flag#3490

Merged
efiop merged 4 commits into
treeverse:masterfrom
Suor:graph-mod
Mar 23, 2020
Merged

dvc: add Repo(skip_checks) flag#3490
efiop merged 4 commits into
treeverse:masterfrom
Suor:graph-mod

Conversation

@Suor
Copy link
Copy Markdown
Contributor

@Suor Suor commented Mar 17, 2020

Will skip modified graph checks on add, run and import. The aim is to
avoid collecting stages and building graph on these commands.

The skip_checks flag is aimed to be private, but let people use dvc on
big repos without monkey patching or writing yaml files themselves.

Closes #2671. For the time being.

@Suor Suor requested a review from efiop March 17, 2020 06:41
Comment thread dvc/repo/__init__.py Outdated
Will skip modified graph checks on add, run and import. The aim is to
avoid collecting stages and building graph on these commands.

The skip_checks flag is aimed to be private, but let people use dvc on
big repos without monkey patching or writing yaml files themselves.
Comment thread dvc/repo/__init__.py
@Suor Suor requested review from efiop and skshetry March 23, 2020 05:24
Comment thread tests/unit/repo/test_repo.py Outdated
@Suor
Copy link
Copy Markdown
Contributor Author

Suor commented Mar 23, 2020

Should be good to go.

Comment thread tests/unit/repo/test_repo.py Outdated
Comment on lines +91 to +94
mocker.patch(
"dvc.repo.Repo._collect_graph",
side_effect=Exception("Should not collect"),
)
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be better to just assert that this wasn't called rather than putting this exception here, as we might have some logic catching this in our code (yes, we don't do that now and catching Exception is a bad thing, but still), which would result in a faulty test. Plus it would be more explicit to use assert not m.called rather than this unnecessary hack.

)
dvc._skip_graph_checks = True

tmp_dir.gen("foo", "foo text")
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also you have dvc_gen above. Not sure if it is intended.

@efiop efiop merged commit a4094f0 into treeverse:master Mar 23, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Optimize stage collection for large repos

3 participants