Incremental processing or streaming in micro-batches

[kskyten]

It seems like it is only possible to replace a dataset entirely and then re-run the analysis. Incremental processing would enable more efficient processing by avoiding recomputation. Here's how Pachyderm does it.

[dmpetrov]

Hi @kskyten
Could you please give us a bit more details? Are you talking about deep learning style computations when a result of each epoch will be saved and then, possible, reused? If so, how do you save the results - as a single file or a set of files in an input directory (like in Pachyderm docs)?

[kskyten]

Let's say you have a folder of images and you define a computationally intensive pipeline for them. After having defined this pipeline you acquire new images that you want to process. This corresponds to the inter-datum incrementality in Pachyderm, if I understood correctly.

[dmpetrov]

We are going to implement Dependency to directory #154 which is a little bit related to this incremental feature. But it is not enough.

This is a good scenario. Let's keep this feature request - it will be implemented. We just need to decide a priority.

@dmpetrov , are we there yet or we should keep this open?

[efiop]

@MrOutis Yep, this should work now by using dvc unprotect: https://dvc.org/doc/user-guide/how-to/update-tracked-files

Closing, please feel free to reopen.

[dmpetrov]

Guys, this is actually a request for streaming\micro-batch processing.

Scenario: I have a dir with images and I'd like to process each image with dwt_ll4.py to generate a spectrum matrixes. If new 23 images were added I'd like to run the script only for these 23 images.

It is mostly data engineering scenario, not data science. So, we should think carefully if DVC should handle this or not. But it is still a valid case and it will be especially useful when we implement multi-processing #755.

So, I'm renaming and reopening.

Thanks, @dmpetrov

[Pierre-Bartet]

Hi, has this eventually been implemented ?

[pared]

Hi @Pierre-Bartet, it have not been yet implemented. Would you find this feature useful? Could you describe your use case?

[Pierre-Bartet]

@pared Use cases equivalent to the one already described:

Scenario: I have a dir with images and I'd like to process each image with dwt_ll4.py to generate a spectrum matrixes. If new 23 images were added I'd like to run the script only for these 23 images.

In fact this is exactly at this point that data version control tools could become really useful because the situation is messier than in the 'vanilla' use case. AFAIK Pachyderm is the only tool with this kind of feature but it is super heavy. I think what is needed is to be able to

• Add an empty collection of objects:
  $ dvc add --collection my_collection

• Create a pipeline using this collection as a dependency, applying computation.py separately on each object of the collection:
  $ dvc run --d my_collection computation.py

• Later add new files to the collection:
  $ dvc add new_file(s) --to my_collection

So that a dvc repro would only run on needed files.

The hard part is that:

$ dvc run --d my_collection computation.py

could both mean:

• 'apply computation.py separately on each object of the collection' -> If for example we want to substract the mean of each image from each pixel
• 'apply computation.py on the whole collection' -> If for example we want to substract the global mean of all the images from each pixel

So I guess the '--d' should be replaced by something else to distinguish between those two cases.