dvc: optimize Git.is_tracked()

[Suor]

Closes #3000.

[efiop]

There might be a new index version released at some point and this might break in a non-obvious way.

[efiop]

Do you you have any benchmark-ish results on how much problems this is causing us right now?

[Suor]

These are not index versions. From BaseIndexEntry.stage docstring:

Stage of the entry, either:

    * 0 = default stage
    * 1 = stage before a merge or common ancestor entry in case of a 3 way merge
    * 2 = stage of entries from the 'left' side of the merge
    * 3 = stage of entries from the right side of the merge

:note: For more information, see http://www.kernel.org/pub/software/scm/git/docs/git-read-tree.html

[efiop]

@Suor How much real-world problems is it causing us? Does it justify this hack at all?

[efiop]

Tbh, glimpsing at it I don't quite understand what is going on.

[Suor]

Benched it - doesn't help much, the reason is .index is not cached it's reread on each use.

[Suor]

Listing 10k files in 10k git repo took 6:19 with old code and 5:16 with new one.

[efiop]

@Suor When do we do that? When we have 10K stage outputs? We don't list files inside dirs, right?

[Suor]

We list when we do dvc add -r dir.

[efiop]

@Suor But we don't recomend doing that on thousands of files, and consider that a misuse. Hence why I'm questioning if we even need this change.

[Suor]

Benchmarked old and new code to walk 10k files in 10k repo, with and without index entries caching:

setting	seconds
old code	379
new code	316
old code + caching	24
new code + caching	19

[Suor]

For 100k files in repo:

100k over 100k	seconds
old code + caching	620
new code + caching	24

All these tests are about running _find_all_targets(dvc, 'dir', True).

[efiop]

@Suor what is index entries caching? Don't see it mentioned/explained anywhere around here.

[Suor]

This is how git.Repo.index property is implemented:

    @property
    def index(self):
        """:return: IndexFile representing this repository's index.
        :note: This property can be expensive, as the returned ``IndexFile`` will be
         reinitialized. It's recommended to re-use the object."""
        return IndexFile(self)

Then IndexFile lazily loads entries and stores them in the corresponding attribute, which doesn't matter since IndexFile is recreated on each access. This looks like a bug in gitpython.

For benching I cached it on our Git class:

    @cached_property
    def index(self):
        return self.repo.index

and then used self.index.entries instead of self.repo.index.entries in Git.is_tracked() - these are lines, which say '... + caching'.

[Suor]

Summary:

• for 10k file repos it's ok to only fix index caching (probably need to fix in gitpython)
• for bigger ones we will need both fixes: the one implemented here and index caching.