diff: clean up output for changed files

[dmpetrov]

Context:

$ dvc -V
0.77.3+e03136
# all txt files are data files
$ ls
newfile.txt     small.txt       ttt.txt
newfile.txt.dvc small.txt.dvc   ttt.txt.dvc

$ echo qwert >> small.txt
$ dvc add small.txt
$ git add small.txt.dvc
$ git commit -m hello
$ dvc diff HEAD^
dvc diff from ada9d97 to 1a22314 .  # <<-- "diff ada9d97..1a22314" might be enough

diff for 'small.txt'
-small.txt with md5 5f73f210b6202aa07279cdff6776b64f
+small.txt with md5 b4301ef665d0343e028fcd5821654edc

added file with size 6 Bytes .  # <<-- File was NOT added, it was modified. 6 bytes is diff, not size.

diff for 'ttt.txt'
-ttt.txt with md5 92a47ab38589702d290647ce24386181
+ttt.txt with md5 92a47ab38589702d290647ce24386181

file size was not changed .  # <<-- what is a reason to output this then? Also, is it about the previous diff or the next one - hard to read?

diff for 'newfile.txt'
-newfile.txt with md5 99370a5386cac00c93c9fdc836076a7d
+newfile.txt with md5 99370a5386cac00c93c9fdc836076a7d

file size was not changed  # <<-- what is a reason to output it then?

Also, it would be great to simplify this output for parsing.

Possible "clean" output:

$ dvc diff HEAD^
diff ada9d97..1a22314
Changed 'small.txt' +6 bytes
md5 5f73f210b6202aa07279cdff6776b64f b4301ef665d0343e028fcd5821654edc
New 'fakefile.txt' +13728494 bytes
md5 92a47ab38589702d290647ce24386181 99370a5386cac00c93c9fdc836076a7d
Removed 'fakefile_2222.txt' +8326436 bytes
md5 e0313660cfc07a10417543b8e4d08bea9 f4daf3bacc9a9494df7d641f572e92b66

Actions:

• Not show not-changed files (at least if it was not specified explicitly).
• Clearly distinguish added files and modified files - with the sizes.
• Improve output for reading. not empty lines within the context of a single file (preferably not empty lines at all). See diff: clean up output for changed files #2982 (comment).
• Simplify the format for parsing. See --porcelain

EDIT 12/21/19:

The same requirements should apply to directories:

$ dvc diff -t dir1 HEAD^
dvc diff from 025a8b3 to 41d8e27

diff for 'dir1'
-dir1 with md5 7492f47e0a1908a80d942a01702e6b8b.dir
+dir1 with md5 8f8824fbd7b1107ef69c85ab5db82973.dir

4 files untouched, 0 files modified, 1 file added, 0 files deleted, size was increased by 37 Bytes

Actions:

• Add an option --porcelain (like in Git) or --to-json to make output format easy-to-parse for scripts.

EDIT 12/24/19:

• Do not print checksum by default. A separate option might be needed.
• Support files inside data directories.

[dmpetrov]

Another request just came #770 (comment) The request is related to data dirs - dvc diff does not go deeper in data dirs. This can be considered as a part or an enhancement of this issue.

Also, it seems like data file checksums should not be shown by default. Let's prioritize human readability like in Git. An option can add checksums.

The output could look like:

$ dvc diff HEAD^
diff ada9d97..1a22314

New:            'fakefile.csv' 751 Mb
New:            'fakefile.txt' 2 Kb

Removed:        'fakefile_2222.txt' 536Kb

Modified:       'small.txt' 36Kb (+6 bytes)
Modified:       'hello.txt' 351 Mb (+12 Mb)
Modified:       'hello.csv' 631 Mb (-3.4 Mb)
Modified dir:   'images'. 116 files untouched, 2 modified, 1 added, 1 deleted. 
      New:         'DM3SZ.jpg` 2.1 Mb
      Modified:    'B8PZM.jpg' 2.5 Mb (+158 Kb)
      Modified:    'T79JK.jpg' 1.7 Mb (-0.2 Mb)
      Removed:     '4USKW.jpg' 2.6 Mb

[jorgeorpinel]

$ dvc diff HEAD^
diff ada9d97..1a22314
Changed 'small.txt' +6 bytes
md5 5f73f210b6202aa07279cdff6776b64f b4301ef665d0343e028fcd5821654edc
...

How about just using the first 7 digits, similar t a short SHA?

md5 5f73f21 b4301ef

or the rel. path in cache

path 5f/73f210b6202aa07279cdff6776b64f b4/301ef665d0343e028fcd5821654edc

[jorgeorpinel]

or nothing (unless a special flag is used), like in the latest example:

$ dvc diff HEAD^
diff ada9d97..1a22314

New:            'fakefile.csv' 751 Mb

Removed:        'fakefile_2222.txt' 536Kb

Modified:       'small.txt' 36Kb (+6 bytes)
Modified dir:   'images'. 116 files untouched, 1 modified, 1 added, 1 deleted. 
      New:         'DM3SZ.jpg` 2.1 Mb
      Modified:    'B8PZM.jpg' 2.5 Mb (+158 Kb)
      Removed:     '4USKW.jpg' 2.6 Mb

Other questions about ^ that example:

• Should there be new lines between New, Modified, and Removed inside directories (indented levels) or vice versa, no new lines in he root level? For consistency
• Should there be a --recursive/-R option with default value of 1 to avoid crazy long diffs? In that case only "files untouched, modified, added, deleted" would be printed for dirs in the last level.
• Should the root diff also print the "files untouched, modified, added, deleted" summary on top? (With this you could use --recursive=-1 to only print the summary, keeping the output super short.)
• I would rec. GNU-style -h human readable file size units (e.g. with humanize.naturalsize(bytes, gnu=True) from this pak.

$ dvc diff -R 2 HEAD^
diff ada9d97..1a22314
0 files/dirs untouched, 2 modified, 1 added, 1 deleted. 
New:            'fakefile.csv' 751 M
Removed:        'fakefile_2222.txt' 536 K
Modified:       'small.txt' 36K (+6 bytes)
Modified dir:   'images'. 116 files/dirs untouched, 7 modified, 1 added, 1 deleted. 
    New:         'DM3SZ.jpg` 2.1 M
    Modified:    'B8PZM.jpg' 2.5 M (+158 K)
    Removed:     '4USKW.jpg' 2.6 M
    Modified dir:   'faces'. 0 files/dirs untouched, 1 modified, 1 added, 0 deleted. 
        New:         'pokerface.jpg` 3.7 M
        Modified dir:   'animals'. 0 files untouched, 0 modified, 100 added, 0 deleted. 
    Modified dir:   'faces'. 3 dirs untouched, 1 modified, 1 added, 1 deleted. 
    Modified dir: ...
    ...

$ dvc diff -R -1 HEAD^
diff ada9d97..1a22314
0 files or directories untouched, 2 modified, 1 added, 1 deleted. 

Also, do we plan to recognize when "1 added, 1 deleted" means that a file was moved, and count it as a single change instead? Like Git does some times. This is probably overkill.

[dmpetrov]

@jorgeorpinel thank you for the questions - all are great points!

• Should there be new lines between New, Modified, and Removed inside directories (indented levels) or vice versa, no new lines in he root level? For consistency

Well... possibly. But instead, I'd remove all the new lines for consistency. See git status might be an example.

• Should there be a --recursive/-R option with default value of 1 to avoid crazy long diffs? In that case only "files untouched, modified, added, deleted" would be printed for dirs in the l ast level.

-R 1 does not look natural. It would be great to see examples from other diff-like commands.

Thank you for bringing this. The problem that I see now - all these spaces as prefixes won't look good at 3rd+ level. We should think about gettig rid of the space-prefixes. It might look like:

$ dvc diff HEAD^
diff ada9d97..1a22314
New:            'fakefile.csv' 751 Mb
Removed:        'fakefile_2222.txt' 536Kb
Modified:       'small.txt' 36Kb (+6 bytes)
Modified dir:   'images'. 116 files untouched, 1 modified, 1 added, 1 deleted. 
New:            'images/DM3SZ.jpg` 2.1 Mb
Modified:       'images/B8PZM.jpg' 2.5 Mb (+158 Kb)
Removed:        'images/4USKW.jpg' 2.6 Mb
New:            'images/outliers/NRD8E.jpg` 1.3 Mb
New:            'images/outliers/W84N2.jpg` 2.0 Mb

• Should the root diff also print the "files untouched, modified, added, deleted" summary on top? (With this you could use --recursive=-1 to only print the summary, keeping the output super short.)

I was thinking about the summary only for data dirs. I don't think it is needed for a regular ones and for dirs inside data dirs (see files in images/outliers/ above - no summary). --recursive=-1 - might be an option - is there any examlpes?

• I would rec. GNU-style -h human readable file size units (e.g. with humanize.naturalsize(bytes, gnu=True) from this pak.

👍 We might show no sizes by default and the numbers with -h.

[jorgeorpinel]

--recursive=-1 - might be an option - is there any examlpes?

My example was

$ dvc diff -R -1 HEAD^
diff ada9d97..1a22314
0 files or directories untouched, 2 modified, 1 added, 1 deleted. 

But you also said

I was thinking about the summary only for data dirs.

Which makes sense, so not sure we want to provide this after all.

-R 1 does not look natural. It would be great to see examples from other diff-like commands.

git diff for example can print very, very long diffs when there's lots of changed files, and it does it without indentation like you suggest (with absolute paths). No recursion/level option is available. (It does have some other options to make the output smaller like --minimal and --name-only though.)

[dmpetrov]

@jorgeorpinel I got your point and -R might be a good option. But I was asking about examples in some popular unix tools like diff, ls, grep, git or similar.

Do you have any examples like this? It's okay if not.

[jorgeorpinel]

Ah no. I don't think -R '-1' exists anywhere haha. I guess it's a little strange. We could also just shift it up to 0 and make the default 1. Just some thoughts about the recursive option but these refinements are probably secondary to the main behavior and could have their own separate issue (lower priority).