Allow quick & shallow copies of Git repos to select which repos to index

[rsdy]

Initiate the clone with:

; curl  -D - "localhost:7878/api/repos/sync?repo=github.com/bloopai/bloop&shallow=true"

This will create a shallow clone (1 commit depth), and a fast index of the repository in question, without looking at the body of any of the files.

After this process is done, the repository ends up in a Shallow state, which means it won't automatically be refreshed until the file filters are configured.

To configure file filters, use the following call:

; curl 'http://127.0.0.1:7878/api/repos/indexed?repo=github.com%2Fbloopai%2Fbloop' \
-X 'PATCH' \
-H 'Content-Type: application/json' \
-H 'Accept: application/json, text/plain, */*' \
--data-binary '{"file_filter": {"rules": [{"include_file": "flake.lock"}, {"include_regex": "^server/"}]}}'

The first call to this will automatically create a Git clone of 1000 commits in depth, and allows syncing the repo fully with the configured file filters.

Note that file filters are additive, and prioritise inclusion over exclusion. In other words, once something's been added to the index, can't be marked to be removed from the index without removing the repository completely!

[gitpod-io]

[anastasiya1155]

I tried it with your example query and also with the reverse one (exclude_file and exclude_regex) and as far as I can see it doesn't index anything by default and only files in 'include_regex' and 'include_file' were indexed. In the case of exclude_file and exclude_regex nothing was indexed at all. Can we change that so that everything is included by default and the user will use 'exclude_' fields to ignore some files and folders?

[oppiliappan]

code changes look good to me, will give this a test.

[ggordonhall]

Works well and code LGTM.

Before merging can you make sure that the commits.rs flow only runs after we've done a deep clone of the repo. As is it panics during the shallow clone step.