repro: use build cache for deterministic stages

[BartekRoszak]

In my experiment I run a few different preprocessing steps which create a different CSV file, then I am modeling this data and also checking different parameters.
When I want to run the same experiment on 4 different machines (dvc is connected to the same remote cache).
Running every type of preprocessing will be done on every machine which takes a lot of time and could be omitted by running dvc pull before dvc repro and dvc push after it.
It could work with one command like dvc repro --remote

[efiop]

Hi @AratorField !

Thank you for sharing your scenario! It would be indeed a very useful feature! I think that dvc checkout would've benefited from it as well. How about something like a autofetch option for dvc config, that will tell dvc that it should fetch cache before performing operations such as repro and checkout? Would that be suitable for you?

Thanks,
Ruslan

[dmpetrov]

@efiop the config option will change API. It won't be clear what exactly dvc repro does from code. An option dvc repro --fetch might be a good option instead of global config param.

[BartekRoszak]

I am running dvc repro in a loop so something like -f -p / --fetch --push will be ok for me.
Setting in config autofetch and autopush is ok too.

[efiop]

@AratorField If you are running it in a loop, why can't you simply call dvc pull before it and dvc push after it? I was thinking about an interactive scenario, where such new options make sense, but in a loop it is easy enough to just call pull/push in your script.

[BartekRoszak]

I am doing it like you said when I run an experiment on a server where I run most of my experiments. But then I am reproducing manually some of the best experiments locally.

[efiop]

Ah, I see. So it looks something like:

git pull
dvc pull
dvc repro
git commit ...
git push
dvc push

Right? If so, I think a more suitable approach would be to add git hooks that will call dvc pull after git pull and dvc push after git push, similar to what we currently have with dvc checkout, where you can call dvc install to install a post-checkout hook for you so dvc checkout is automatically called after git checkout. Would that suit you?

Currently dvc install only installs post-checkout hook, so we will have to add support for post-pull and post-push hooks as well. Which totally fits within our current API architecture.

[BartekRoszak]

Oh... So it works totally differently than I thought. Now I know why my algorithm is doing preprocessing every time.

Let me explain you a bit more.
I am building about 100 models with different parameters (standard hyperparameter optimization).
Before data modeling there is preprocessing, so in dvc repro there are two steps, preprocessing and modeling.
I am working with text and in preprocessing I have script which can do two things:

1. convert all to lowercase
2. remove punctuation

So there could be 4 different types of preprocessing:

• None
• convert all to lowercase
• remove punctuation
• convert all to lowercase and remove punctuation

I am setting the type of preprocessing in a config file which is a dependency of preprocessing.py which is in pipeline.

What I thought the DVC is working is:
After running dvc repro with some type of preprocessing and then run dvc repro again with another type of preprocessing, it won't do preprocessing again if I run it for the third time with the first type of preprocessing because DVC will say:

Hay! We have already done this type of preprocessing and stored the output file in the cache so now I can just take this file from the cache instead of running preprocessing again!

But it looks like it stores only the last version of the file, right?

[efiop]

You are right, dvc doesn't currently remember every dependencies + command = outputs combination that have occurred in the past, so it can't just pull up outputs from the cache this way. That being said, I have been thinking about this scenario a lot in the past and it would be indeed extremely useful to have such feature. The best way we could implement this is to utilize git(or any other scm system that the repository is based on) to get a table of dependencies + command = outputs values, that will help us quickly identify if this combination of dependencies has been already processed by this particular command, so we could pull up appropriate outputs without recomputing. I would call this something like a "build cache" as apposed to the current "data cache" and would indeed utilize it with a special option for dvc repro, e.g. something like dvc repro --use-build-cache that will tell dvc to check every stage to see if this combination(of deps and command) has been built before and if it was then it is okay for dvc to just pull up the result without recomputing. It is also worth pointing out that since dvc doesn't keep 100% track of the environment you run your pipeline in(e.g. system libs versions and so on), there is always a chance that you won't get the same result if you build something in two environments. That said, with a little bit of care from the user(e.g. probabilistic models will always produce different result, so if you were to use --use-build-cache option, you need to be aware of that it will pull up the old result which will not be equal to the one that you would've got by actually rebuilding) , this feature should be extremely useful. Is this something that you would be interested in? If you are interested, we can up the priority for this one.

[BartekRoszak]

It would love to see it!
As you said, one has to be careful because not every script is deterministic, so maybe a better way to do that is to add an adequate option to dvc run like --deterministic. So when you will run dvc repro it will know which part of the pipeline could be taken from the cache.

[efiop]

Makes sense! I will look into it soon. Thank you so much for the feedback!

[efiop]

NOTE: we can use git history as build cache by searching for existing committed dvc files in history. We could also cache that operation to only parse git history once. Also need to store local build cache for uncommited changes. Let's start with the latter one.