[WIP] dependency: fine grained (user cmd filter)

[casperdcl]

• modify schema
  • dvc.yaml
  • dvc.lock
• support running (from dvc.yaml:stages.<stage>.deps.[].<path>.cmd)
• add to dvc.lock
• add way to add user cmd filter via CLI API
  • maybe dvc run -d "utils.py:extract_function.py --name check_db"
• add tests
• fixes Support function specific dependencies #3439

Note that this implementation

• uses PARAM_FILTER = "cmd"
• passes path as a positional argument to the user-defined cmd
• computes the md5sum of the user-defined cmd output
• only works on local files (not dirs & not remote paths)
• only works for dependencies (not outputs)
• would produce the old behaviour (whole-file hash) if setting cmd: cat
• was tested using https://github.com/casperdcl/dvc-udf

schema:

# in dvc.yaml
utils.py:
  cmd: python extract_function.py --name check_db

# in dvc.lock
path: utils.py
cmd: python extract_function.py --name check_db
md5: s0m3h45h # computed via `{cmd} {path} | md5sum`

testing:

git clone https://github.com/casperdcl/dvc-udf
cd dvc-udf
pip install -r requirements.txt
dvc repro -f -v out  # assuming dvc installed from this PR

[casperdcl]

not sure if there are aspects of stage.run.cmd_run which should be used here

[casperdcl]

not sure if this is required - maybe automatically handled elsewhere (i.e. entire tmpdir deleted before exit)

[casperdcl]

maybe assert not required (should be handled by schema)?

[skshetry]

Other CLI commands can create/load dependencies, skipping the schema. Good to have an assert.

[efiop]

Closing for now, we'll get back after dep/out refactor to properly accommodate this.