Weekly Digest (12 April, 2020 - 19 April, 2020)

[weekly-digest]

Here's the Weekly Digest for iterative/dvc:

---

ISSUES

Last week 24 issues were created.
Of these, 13 issues have been closed and 11 issues are still open.

OPEN ISSUES

💚 #3648 Restyle remote: add support for WebDAV, by restyled-io[bot]
💚 #3647 remote: add support for WebDAV, by shizacat
💚 #3645 Fix repro duplicating effort while checking steps #3644, by charlesbaynham
💚 #3644 repro -P seems to check status of dependancies too often, by charlesbaynham
💚 #3643 Unclear dependencies conflict when installing from conda-forge, by drorata
💚 #3642 Consistent cmd option order?, by jorgeorpinel
💚 #3641 params: update bash and zsh scripts with the latest commands, args, by shcheklein
💚 #3636 dvc pull with target ignoring specific targets inside, by HectorAnadon
💚 #3635 remote: loading dirs is exteremely slow because of PathInfo, by efiop
💚 #3634 [WIP] remote: locally index list of checksums available on cloud remotes, by pmrowla
💚 #3633 Pipeline variables from params file, by dmpetrov

CLOSED ISSUES

❤️ #3649 tests: test_directories, by nik123
❤️ #3646 Restyle WIP: Fix repro, by restyled-io[bot]
❤️ #3640 tests: add some DvcTree unit tests, by efiop
❤️ #3639 dvc: introduce DvcTree, by efiop
❤️ #3638 Restyle remote config validation, by restyled-io[bot]
❤️ #3637 remote: don't use PathInfo.from_posix, by efiop
❤️ #3632 remote: use .dir checksum existence to infer file contents existence, by pmrowla
❤️ #3631 params: diff: introduce --all, by efiop
❤️ #3630 stage: resolve stage path before creating the stage, by efiop
❤️ #3629 Error in dvc pull due to python encoding, by vss96
❤️ #3628 remote config validation, by karajan1001
❤️ #3626 Specify cmd arg names (metavars), et al., by jorgeorpinel
❤️ #3625 dvc: meaningful option arguments, by jorgeorpinel

LIKED ISSUE

👍 #3632 remote: use .dir checksum existence to infer file contents existence, by pmrowla
It received 👍 x4, 😄 x0, 🎉 x2 and ❤️ x2.

NOISY ISSUE

🔈 #3626 Specify cmd arg names (metavars), et al., by jorgeorpinel
It received 4 comments.

---

PULL REQUESTS

Last week, 18 pull requests were created, updated or merged.

UPDATED PULL REQUEST

Last week, 5 pull requests were updated.
💛 #3648 Restyle remote: add support for WebDAV, by restyled-io[bot]
💛 #3645 Fix repro duplicating effort while checking steps #3644, by charlesbaynham
💛 #3634 [WIP] remote: locally index list of checksums available on cloud remotes, by pmrowla
💛 #3598 config: forbid setting core.no_scm outside repo, by mroutis
💛 #3577 Metrics - plotting for multiple revisions initial, by pared

MERGED PULL REQUEST

Last week, 13 pull requests were merged.
💜 #3649 tests: test_directories, by nik123
💜 #3640 tests: add some DvcTree unit tests, by efiop
💜 #3639 dvc: introduce DvcTree, by efiop
💜 #3637 remote: don't use PathInfo.from_posix, by efiop
💜 #3632 remote: use .dir checksum existence to infer file contents existence, by pmrowla
💜 #3631 params: diff: introduce --all, by efiop
💜 #3630 stage: resolve stage path before creating the stage, by efiop
💜 #3628 remote config validation, by karajan1001
💜 #3626 Specify cmd arg names (metavars), et al., by jorgeorpinel
💜 #3624 Updates latest version of PyDrive2, by mribeirodantas
💜 #3619 refactor: extract stage's relationship with Dvcfile out from Stage, by skshetry
💜 #3595 scm: git: clone: add progress, by casperdcl
💜 #3584 dvc: implement multi-stage dvcfile, by skshetry

---

COMMITS

Last week there were 13 commits.
🛠️ tests: test_directories (#3649) Fixed test_directories test in tests/func/test_diff.py by nik123
🛠️ [Specify cmd arg names (metavars), et al. (#3626) * add: -f arg name from FILE to
to match dvc.org cmd ref

• dvc: cmd arg names (metavars) to match docs
  for Specify cmd arg names (metavars), et al. #3626

• metrics: update help output to indicate supported types
  per Metrics: separating scalar and continuous/graphs metrics #3409
  and for Support yaml as metrics file #3572

• dvc: reorder subcommands to match docs
  for Consistent subcommand order #3608](14b213a) by jorgeorpinel
  🛠️ [scm: git: clone: add progress (scm: git: clone: add progress #3595) * scm: git: clone: add progress

• git: add progress desc

• wip

• git: tidy message](f0c81bc) by casperdcl
  🛠️ [dvc: add a support for multistage dvcfile. (dvc: implement multi-stage dvcfile #3584) Now, pipeline stage files can house multiple stages, and separate lockfiles
  are created which has the checksums, whereas Dvcfile will be clean and human
  readable and editable. The *.dvc files will be generated for output files.

It is available via a hidden flag: -n.

Fixes #1871
PR: #3584](128fa7f) by skshetry
🛠️ tests: add some DvcTree unit tests (#3640) by efiop
🛠️ dvc: introduce DvcTree (#3639) This is useful as it is in dvc ls implementation, but this is also a
prerequisite for RepoTree that will combine git(workspace) files and
dvc files into one tree, allowing things like dvc metrics and
external_repo to have a universal interface to access whatever
file/dir they want without cumbersome git/dvc path detection. by efiop
🛠️ [remote: don't use PathInfo.from_posix (#3637) Using it to load dirs takes around 50 sec compared to 0.4 with simple
replace(). Path objects are great, but not when you need lots of them.

Related to #3635](45c2c72) by efiop
🛠️ [remote config validation (#3628) * remote config validation

Fixes #3552

• solving config test fail

• Validation problem in config merging

sloving validation problem in config merging from different levels.
Only those config in lower levels can affect the validation process.

• reduce Cognitive Complexity

• Two request changes

1. Duplicate load current level of config
2. rewrite test to pytest style

• black it

• config: don't use all_levels

Co-authored-by: karajan1001 jah@mail.ustc.edu.cn
Co-authored-by: Ruslan Kuprieiev ruslan@iterative.ai](d3d471c) by karajan1001
🛠️ [params: diff: introduce --all (#3631) * params: diff: introduce --all

Makes dvc params diff also show unchanged params.

• params: diff: show all present parameters](8455b7b) by efiop
  🛠️ [remote: use .dir checksum existence to infer file contents existence (remote: use .dir checksum existence to infer file contents existence #3632) * repo: separate dir cache and file cache in memory

• used_cache()/get_used_cache() in repo/stage/output now return
  tuples of (dir_cache, file_cache) instead of one flat/merged cache

• update tests for new get_used_cache behavior

• remote: if .dir checksum exists on remote, assume contents also exists

• affects all commands which use cache_exists() (remote status)

• push: only upload .dir file after full file contents has been uploaded

• gc: always remove .dir checksums first

• functional tests for push/gc

• repo: support nesting dir caches in NamedCache

• remote: NamedCache updates

• Fix tests

• Fix deepsource warnings

• tests: use pytest mocker fixture

• Update dvc/cache.py

Co-Authored-By: Saugat Pachhai suagatchhetri@outlook.com

Co-authored-by: Ruslan Kuprieiev kupruser@gmail.com
Co-authored-by: Saugat Pachhai suagatchhetri@outlook.com](7275843) by pmrowla
🛠️ [refactor: extract stage's relationship with Dvcfile out from Stage (#3619) * refactor stage into dvcfile

• Split create_stage out of dvcfile to Stage::create() and overwrite

• Remove logging from Repo::run

• tests: stop using --remove-outs

Co-authored-by: Ruslan Kuprieiev ruslan@iterative.ai](65854c4) by skshetry
🛠️ Updates latest version of PyDrive2 (#3624) This latest version of PyDrive2 fixes an issue for Google Drive remote management in machines with an old version of httplib2. by mribeirodantas
🛠️ [stage: resolve stage path before creating the stage (#3630) Current logic is extremely complex and messy, hiding a lot of bugs
inside. There are pretty much only 2 scenarios for stage creation:

1. dvc add/import/etc when dvcfile is created beside the output file
2. dvc run when dvcfile is created at the place of execution

both are transparently affected by --fname flag. Since we've removed
--cwd complexity already, there is no reason to generalize stage path
resolution for those two scenarios.](fc6a10f) by efiop

---

CONTRIBUTORS

Last week there were 8 contributors.
👤 nik123
👤 jorgeorpinel
👤 casperdcl
👤 skshetry
👤 efiop
👤 karajan1001
👤 pmrowla
👤 mribeirodantas

---

STARGAZERS

Last week there were 50 stagazers.
⭐ WKSu
⭐ vss96
⭐ AlexxNica
⭐ diegombar
⭐ danieldsf
⭐ githubassets
⭐ Ly0n
⭐ colour-me-bibi
⭐ its-a-magical-world
⭐ khklau
⭐ DanielGutmann
⭐ allforabit
⭐ rastogin-rp
⭐ gwater
⭐ johnclaus
⭐ chiragjn
⭐ EthenZhang
⭐ peleplay
⭐ caobinbin
⭐ markusmaresch
⭐ keeganmccallum
⭐ erliding
⭐ HiromuHota
⭐ ga29fol
⭐ latinojoel
⭐ nandesh553
⭐ bvezilic
⭐ svlandeg
⭐ garciparedes
⭐ vladbash
⭐ 26keisuke
⭐ yerdaulet1991
⭐ bscherrer
⭐ arocketman
⭐ vikashsalvi
⭐ ivanmarkov97
⭐ amanvishnani
⭐ cmealo
⭐ bohblue2
⭐ kibernick
⭐ RamanHacks
⭐ ali-h
⭐ chrisshaw
⭐ pvictor
⭐ hydr063n
⭐ chriskalmar
⭐ ChristianGerloff
⭐ sergeyt
⭐ dguera
⭐ nik123
You all are the stars! 🌟

---

RELEASES

Last week there were no releases.

---

That's all for last week, please 👀 Watch and ⭐ Star the repository iterative/dvc to receive next weekly updates. 😃

You can also view all Weekly Digests by clicking here.

Your Weekly Digest bot. 📆