Support function specific dependencies

[drorata]

Assume we have 4 stages:

• Stages 0 and 2 depend on the function foo
• Stages 1 and 3 depend on the function bar

In order to reduce code duplication we want to put the utilities foo and bar in one place; let them be in utils.py. So, utils.py becomes a dependency of all four stages.

Now... after some time we realize we have to update something in bar. So we edit utils.py. This, in turn, invalidate all stages. Obviously, a simple solution is to have utils_0_2.py which will be a dependency of 0 and 2 and similarly utils_1_3.py. But this solution conflicts with good practices of coding.

A possible solution could be to adopt some notation like the one used in pytest and specify the function inside utils.py. For example, something like:

deps:
- md5: 409a438ee6ac70fc1f911631412325ea
  path: utils.py::foo

This is a continuation of this discussion.

[pared]

Ok, so first obstacle I see is that we are trying to make dvc language agnostic, so the specification of how to define this function dependency and how to calculate its checksum would have to be quite abstract. Having said so, it still looks like reasonable feature request.
@iterative/engineering what do you think?

[casperdcl]

We'd need specific per-language support (based on file extension) to extract source code of functions/classes and md5 hash them. Python would be a first obvious language to support but in general we'd need one plugin/parser per lang.

[shcheklein]

Alternative to this is to go all Python. We've seen examples when companies use DVC core API to manipulate stage files from Python. It usually means that users do not use dvc run, etc. They use some custom decorators that internally handle stage files, etc, etc.

[casperdcl]

Not sure whether going "all Python" would be a different implementation than supporting other langs. We'd probably still have to check file extensions/types and print out errors if it's a binary file etc...

[jorgeorpinel]

Note: this may be a duplicate of #1577.

@casperdcl I think Ivan means an alternative for @drorata is to integrate with the DVC using the Python package (pip installation) instead of using dvc run and CLI in general. We don't have any docs about it yet though (see treeverse/dvc.org/issues/998), and docstrings are sometimes very complete, sometimes inexistent (see #3176), so that's probably a corner solution for rather advanced users.

[casperdcl]

Well we could support a user-defined external filter tool (similar to git):

deps:
- md5: 409a438ee6ac70fc1f911631412325ea
  path: utils.py
  filter: python extract_function.py --name foo

[fabiosantoscode]

I agree with @casperdcl here. I don't think we should be maintaining a whole bunch of tools for this. Maybe we can write just one, for python, and distribute it on pypi to "set the tone" for others to provide other tools with similar arguments (and possibly command name prefixes like dvc-extractfn-python --name foo).

Users of other languages will want to distribute this as a package in their own language. And as long as it's in the PATH and has the same signature, we're fine with it.

Edit: same argument spec and same command prefix aren't mandatory but sure would be nice for user experience.

[fabiosantoscode]

If anyone in the world wants one of these for JavaScript I can write it, it should be 50 lines or less using a pre-existing parser.

[elgehelge]

Are you sure you really want such detailed dependency management @drorata ? Will you remember to update the dependency when somebody comes along and for instance splits the function into two?

def foo(bar):
    return [ba + 1 for ba in bar]

to

def foo(bar):
    return [fo(ba) for ba in bar]

def fo(ba):
    return ba + 1

Do you have an example of "this solution conflicts with good practices of coding" to make it a little less abstract?

[casperdcl]

@elgehelge good point, but I think it's fine for dvc to give users double-edged swords with caveat emptor warnings. If we go down the "support user-defined filter: " route then users could define a really clever filter which even analyses the whole code base and pulls in a function's dependencies. Probably not too hard considering:

it should be 50 lines or less using a pre-existing parser

I see this as part of DVC's "Makefiles-but-better" bit, where there's sub-file granularity.

The question is, does this feature make bad practice so easy for end-users to slip into that DVC should stay away from it? I don't think so.

If nothing else the end-user filter could be a simple dos2unix | sed -i '/^$/d' to minimise re-running things after whitespace changes...

btw I assume filter is a POSIX-style pipe, i.e. cat path | filter | md5sum is the md5 saved in the DVC-file.

CC @efiop