dvc metrics diff

[dmpetrov]

Today, we can track metrics but metrics become much more valuable when you can see differences/improvements over time (commits/branches). A new dvc metrics diff command is needed.

$ dvc metrics diff HEAD^^
        metr.json:
		{
		    "top1-error": 0.0385,
		    "top5-error": 0.039221
		}

Open question: What should we do about not float/integer metrics? Let's don't support (ignore) them. Any other ideas?

Note, we should deal with float formatting carefully - we don't want to see diff values like 0.0001624000000000000001.

Also, an easy-to-parse output option is required:

$ dvc metrics diff HEAD^^ --to-json
{
    {
        "file": "metr.json"
	"changed": {
            "top1-error": {
                "old": 0.0385,
                "new": 0.0384824,
                "diff": 0.0000176
            },
        "removed": {
            "top5-error": {
                "old": 0.039221
            }
        }
        "added": {
            "loss": {
                "new": 0.0384824
            }
        }
}

[efiop]

A workaround would be pretty trivial though. dvc get from HEAD^^, dvc get from current HEAD and run old plain diff on it.

[dmpetrov]

@efiop yeap! Plain diff won't work, unfortunately - the order of metrics might easily change and you see a total mess. Also, we need a nice looking diff numbers, not old and new values.

But agree, we just need a nice looking shortcut for that.

[DavidGOrtega]

Are you guys contemplating in the specs every stage of the pipeline?
Something like this:

{
      "train": {
        "train_time": "3d 8h 23m 15s",
        "memory_consume": "8Gb"
      },
      "eval": {
        "inference_time": 0.001,
        "memory_consume": "124Mb",

        "top1-error": 0.0385,
        "top5-error": 0.039221
      }
    }

[pared]

@dmpetrov

Open question: What should we do about not float/integer metrics? Let's don't support (ignore) them. Any other ideas?

For starters, I would not support them, but I think at some point we could add functionality allowing the user to define how to calculate the difference between particular metrics, by executing some custom-defined method. That could resemble how we are dealing with summon right now.
It sounds kind of complicated as for now.

Question from me:
Do we want to support diff only for 2 revs ("old" and "new")? Or will we want to support revs ranges at some point (eg dvc metrics diff HEAD~10 for observing changes during last 10 iterations)? If so, the notion of "old", "new" and "diff" might have to be changed.

[dmpetrov]

That could resemble how we are dealing with summon right now.

👍 I have the same thoughts - we need a "custom" metrics file to support multiple formats (our basic json + csv are not enough). Probably, different types of metrics can be represented as separate summoning-objects (not necessary metrics inside summoning object).

Metrics are a bit more complicated than just numbers. There are numbers, numbers with сonfidence level, confusion matrixes and so one. It is not easy to find a single way of dealing with them. Even for numbers metrics, it is important to know if you are minimizing or maximizing it (is +013 good or bad?) to visualize properly and find the "best one".

Do we want to support diff only for 2 revs

Only two. The same idea as git diff.

[DavidGOrtega]

Question from me:
Do we want to support diff only for 2 revs ("old" and "new")? Or will we want to support revs ranges at some point (eg dvc metrics diff HEAD~10 for observing changes during last 10 iterations)? If so, the notion of "old", "new" and "diff" might have to be changed.

One of the best features would be observability. One of the best features of dvc is having all the metrics all together. I would vote for having that somehow.

[pared]

@DavidGOrtega @dmpetrov Well, that seems like a conflict "behave as git" vs "this might be useful in ML use-case". I guess we can always think about implementing another new metrics command (--range flag for diff could also be option, but I think it would not make too much sense, if we are going to stick to old, new notion for diff)

[dmpetrov]

@DavidGOrtega I'd appreciate if you could provide a use case (with a command example) when it will be helpful (and not easily replicable with multiple dvc diff)