diff: reimplement interface and tests from scratch

[ghost]

Close: #2982, #2998, #2999, #1893, #2372
Related: #770

• Shows the difference between HEAD and the current workspace by default
• Don't show files that stayed the same between both trees.
• Remove --target option.
• Checksums are only displayed when --checksum option is provided.
• Format diff for machine readability when --show-json option is given.
• Changes in output:
  • Remove file size information
  • Recurse directory outputs and display its cached information.
  • Group diff by state (i.e. "Added", "Deleted", "Modified")
• Allow running dvc diff when there's no cache available.
• Update completion scripts
• Update documentation (diff: update docs according to the new patch dvc.org#953)

Questions:

• Should we introduce a --recursive option to explicitly recurse directory outputs?
• Should we support renamed files?

[efiop]

@MrOutis What is the point of these tests if they are all commented out and TODos?

[efiop]

As for the CLI, I was thinking on using the following as output:
Would it be OK to show checksums only when --checksums (new option) is specified?

I think the ticket was saying that we shouldn't show checksums by default. Wouldn't bother with introducing --checksums until someone asks. It should be present in json output though.

[Suor]

Looks cool so far.

@Suor, could you please take a look again? I did quite a lot of changes)

[Suor]

The order of dict keys are not preserved before Python 3.6, so in Python 3.5 the sections will go in random order.

[ghost]

that's why the py3.5 test suite was failing! 🙈

[ghost]

now iterating with an explicit order

[Suor]

You are abusing set mechanics and __eq__ here and below, this results into iterating over intersection very tricky and unobvious. People reading this code will expect it to conform to:

old - (old ^ new) == new - (old ^ new) == old & new

which it doesn't. And we don't need this anyway as we may simply use dicts:

saved_tree = repo.tree
try:
    repo.tree = get_ree(a_ref)
    # we don't have repo.outs, but might be we should, 
    # or make it repo.iter_outs()?
    old = {str(out): out.checksum for out in repo.outs}
    repo.tree = get_ree(b_ref)
    new = {str(out): out.checksum for out in repo.outs}
finally:
    repo.tree = saved_tree

# set efficiently converts dict keys to set, we only compare names here
added = {name: new[name] for name in set(new) - set(old)}
deleted = {name: old[name] for name in set(old) - set(new)}
changed = {
    name: {"old": old[name], "new": new[name]} 
    for name in set(old) & set(new) 
    if old[name] != new[name]
}

[ghost]

this was mind-bending 👌 thanks for pointing it out, @Suor !
just added it)

[ghost]

however, instead of repo.iter_outs, I defined a method to return the dictionary with paths and checksums, might refactor that one later

[dmpetrov]

I've done testing. New issues were created for features outside of the scope of this change: #3256, #3255. Great communication during the process!

LGTM 👍

[efiop]

@MrOutis please check the tests, looks like there are some dict ordering issues.

[ghost]

@efiop , doing it right now

@Suor, I changed the diff logic to use dicts

[jorgeorpinel]

What's the fastest way to see if this is released or get notified when it is? I don't think it's included in 0.82.6, for example. Thanks