Reduce Repo Size

[zachbogart]

I noticed the repo is really big to clone every time, but the size of the files are not that large. Found that all of the space is taken up by the .git folder: all the git stuff. We should look into cleaning the repo and making it smaller.

As a reference, our .git folder is ~700MB, while r4ds is ~30MB.

Here's some relevant reading. We should discuss this at your office.

https://stackoverflow.com/questions/28720151/git-gc-aggressive-vs-git-repack
https://git-scm.com/book/en/v2/Git-Internals-Maintenance-and-Data-Recovery
https://stackoverflow.com/questions/2116778/reduce-git-repository-sizehttps://stackoverflow.com/questions/1398919/make-git-consume-less-disk-space

[zachbogart]

This may prove helpful:
http://stevelorek.com/how-to-shrink-a-git-repository.html

[jtr13]

Thanks! I noticed it was big to clone but didn't realize that the .git folder was the issue; I assumed it was because we have so many graphs. Will look into this.

[jtr13]

18F/C2#439

How to get rid of large .git objects pack files (ours is 733MB!!!!)

[zachbogart]

Started R community question:
https://community.rstudio.com/t/reduce-github-repo-size-of-a-bookdown-project/18319

[jtr13]

This might be helpful: https://help.github.com/articles/removing-sensitive-data-from-a-repository/
Curious if git ls-files shows that you're still tracking files/folders that are in .gitignore now.

[zachbogart]

I tried following this method (18F/C2#439) and pushing the result, but I don't get different behavior when I try to clone a new copy.

By the way, you should probably clone a new copy to make sure everything agrees if you want to make changes.

[zachbogart]

Used bfg and it looks like it worked. Your Joyce branch is quite behind now I think. Got it down to around 18MB.

The bad news is this rewrote history, which means people with open PRs have to clone a fresh copy and add their changed files. I will try to monitor the .git size and see if it still grows or if it was due to earlier work piling up.

[jtr13]

That's fantastic! I don't see any open pull requests (how did you manage that?)... do you mean ppl who have already forked the repo and are planning to contribute but haven't submitted a pull request yet?

[jtr13]

Ok, I deleted the Joyce branch... it seemed hopeless to update it and it's not good practice to keep an open branch like that anyway--it's better to start a new branch for each new contribution. I deleted everything locally (burn it down) and recloned. Now it's 39MB. Not sure what caused the doubling but a lot lot lot better than before.

[zachbogart]

People who had open PRs need to create new ones with the newest repo.

That increase in size is a little worrying.

[jtr13]

Ok got it. The culprit is a 17MB pack file. I wonder if the Joyce branch somehow caused it since it was linked to the large repo... maybe rewrite the history again with all references to that branch deleted? Or maybe there's a direct way to delete that pack file.

[zachbogart]

I feel like the cleaning of files worked, but it should be done periodically when no PRs are open (probably between semesters) to keep the size down while reducing the fallout of people having to update their forks.