Run git gc periodically on the crates.io index (#778) by ohaddahan · Pull Request #956 · rust-lang/docs.rs

ohaddahan · 2020-08-11T19:16:23Z

Forking git gc periodically from start_registry_watcher.

Closes #778.

Co-authored-by: Chase Wilson <buckshot1233@gmail.com>

… rails with "unused static"

Nemo157 · 2020-08-12T07:32:29Z

+use std::time::{Duration, Instant};
+
+fn run_git_gc() {
+    let gc = Command::new("git").args(&["gc", "--auto"]).output();


This needs the path to the repo to clean. I think it'd make most sense to move this to a method on index::Index since that's what needs cleaning and has the path available. You can then access it via docbuilder.index to call.

@Nemo157 inside what method?

How about RustwideBuilder::build_package ? Doesn't it run in the same location as the git repo of the crate?

A new method, probably just keep this name for it run_git_gc.

I have vague ideas of eventually removing the index to queue handling from docbuilder completely, it isn't really related to building the docs.

@Nemo157 what I meant was, where in the Index flow to add it.

I checked now and the Index is called from RustwideBuilder which is called from the Queue.

So maybe add last_gc_run property on the Queue and call run_git_gc conditionally from inside build_package? Something along these lines?

I was thinking just keep calling it from here, this is the place that causes us to create new packfiles so should be responsible for knowing when to clean them up.

@Nemo157 what is the location you refer to by calling it from here?

I assume he meant daemon.rs. So fn run_git_gc gets moved to Index, but still gets called from start_registry_watcher.

@jyn514 @Nemo157 updated, let me know if that what you meant 😊

Pre commit hook

… and CURRENTLY_RUNNING_THREADS declaration match the usage, "target_os = linux"

jyn514 · 2020-08-12T23:04:15Z

+use std::time::{Duration, Instant};
+
+fn run_git_gc() {
+    let gc = Command::new("git").args(&["gc", "--auto"]).output();


I assume he meant daemon.rs. So fn run_git_gc gets moved to Index, but still gets called from start_registry_watcher.

Kixiron · 2020-08-13T15:26:56Z

@ohaddahan Could you push your commits so we can see the changes?

Pre commit hook

ohaddahan · 2020-08-13T16:47:29Z

@Kixiron ops sorry, forgot to merge my fork branch 😊

Co-authored-by: Chase Wilson <buckshot1233@gmail.com>

… rails with "unused static"

… and CURRENTLY_RUNNING_THREADS declaration match the usage, "target_os = linux"

Co-authored-by: Chase Wilson <buckshot1233@gmail.com>

jyn514 · 2020-08-13T21:02:08Z

@ohaddahan Hi, your commits messages say 'rebase' but rebases do not have commits, they apply the original commits on top of the original branch. As it stands this has 143 commits and a 900 line diff. Can you please rebase using something like the following:

git checkout master
git fetch origin
git rebase -i origin/master

Since you have several merge commits there will be many conflicts, so you should drop all commits that are not related to this PR. Alternatively, it might be easier to start over on a new branch:

git checkout origin/master
git checkout --branch git-gc
git cherry-pick all the relevent commit hashes

However, as-is, this is very difficult to review.

jyn514 · 2020-08-13T21:02:35Z

If you want help feel free to join the discord channel: https://discord.gg/f7mTXPW

ohaddahan · 2020-08-14T06:54:35Z

@jyn514 yeah my fork went down the drain, I'll clean it up.

jyn514 · 2020-08-15T16:19:03Z

Closing in favor of #975

Ohad Dahan and others added 3 commits August 11, 2020 22:09

Run git gc periodically on the crates.io index #778

69e51b4

Update config.rs

9cb547f

Fixed "cargo fmt"

3d0ec4e

Kixiron reviewed Aug 11, 2020

View reviewed changes

Comment thread .gitignore

Comment thread src/config.rs Outdated

Comment thread src/utils/daemon.rs Outdated

Comment thread src/utils/daemon.rs Outdated

Comment thread src/utils/daemon.rs Outdated

ohaddahan and others added 9 commits August 11, 2020 23:22

Update src/utils/daemon.rs

ce5bbcf

Co-authored-by: Chase Wilson <buckshot1233@gmail.com>

Update src/utils/daemon.rs

94f8ac3

Co-authored-by: Chase Wilson <buckshot1233@gmail.com>

Update src/utils/daemon.rs

6335dd5

Co-authored-by: Chase Wilson <buckshot1233@gmail.com>

Update src/config.rs

d3e2c80

Co-authored-by: Chase Wilson <buckshot1233@gmail.com>

Update .gitignore

8d06f51

Co-authored-by: Chase Wilson <buckshot1233@gmail.com>

Added --auto

e323a2f

--auto and fmt

6d66f9e

Fix cargo fmt

29b1113

Adding pre-commit hook, adding dead-code lint since without it clippy…

014338c

… rails with "unused static"

Nemo157 reviewed Aug 12, 2020

View reviewed changes

emilyalbini reviewed Aug 12, 2020

View reviewed changes

Comment thread src/config.rs Outdated

jyn514 added A-admin Area: Administration of the production docs.rs server S-waiting-on-author Status: This PR is incomplete or needs to address review comments labels Aug 12, 2020

Ohad Dahan and others added 2 commits August 12, 2020 19:59

Changed envar prefix and replace echo with printf for pre-commit hook

45892ff

Merge pull request #1 from ohaddahan/pre-commit-hook

525c6f8

Pre commit hook

ohaddahan commented Aug 12, 2020

View reviewed changes

Comment thread src/web/metrics.rs Outdated

ohaddahan commented Aug 12, 2020

View reviewed changes

Comment thread git_hooks/pre-commit

Kixiron reviewed Aug 12, 2020

View reviewed changes

Comment thread src/web/metrics.rs Outdated

Comment thread src/web/metrics.rs Outdated

Removed allow(dead_code) and not(windows) , now OPEN_FILE_DESCRIPTORS…

1e4d83e

… and CURRENTLY_RUNNING_THREADS declaration match the usage, "target_os = linux"

jyn514 reviewed Aug 12, 2020

View reviewed changes

Ohad Dahan added 3 commits August 13, 2020 15:57

Updated pre-commit script and instructions

39e5b26

Added newline at end of pre-commit script

3be80f5

Move run_git_gc to Index

1ba6fd5

Merge pull request #2 from ohaddahan/pre-commit-hook

7f09aea

Pre commit hook

ohaddahan and others added 22 commits August 13, 2020 22:42

Update src/config.rs

28065e1

Co-authored-by: Chase Wilson <buckshot1233@gmail.com>

Update .gitignore

be9e3e0

Co-authored-by: Chase Wilson <buckshot1233@gmail.com>

Fix cargo fmt

89578dd

Adding pre-commit hook, adding dead-code lint since without it clippy…

87ed97d

… rails with "unused static"

Changed envar prefix and replace echo with printf for pre-commit hook

2bf3ae3

Removed allow(dead_code) and not(windows) , now OPEN_FILE_DESCRIPTORS…

a9efd05

… and CURRENTLY_RUNNING_THREADS declaration match the usage, "target_os = linux"

Updated pre-commit script and instructions

d7c0ec1

Added newline at end of pre-commit script

a7568a2

Move run_git_gc to Index

db8ce50

Fix typo

4121fb6

Rebased

c7fcff6

Update .gitignore

d8b8a75

Co-authored-by: Chase Wilson <buckshot1233@gmail.com>

Rebased

60c58fb

Changed envar prefix and replace echo with printf for pre-commit hook

3ec3b83

Rebased

79a295d

Added newline at end of pre-commit script

02bc0ff

Rebased

ff02d23

Rebased

5722ae1

Rebased

6c7193b

Rebased

ef9cd22

Rebased

a593212

Rebased

5b0cc9f

This was referenced Aug 15, 2020

pre-commit hooks #973

Merged

Fix readme #974

Merged

Run git gc periodically on the crates.io index #975

Merged

jyn514 closed this Aug 15, 2020

Conversation

ohaddahan commented Aug 11, 2020 • edited by jyn514 Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Kixiron commented Aug 13, 2020

Uh oh!

ohaddahan commented Aug 13, 2020

Uh oh!

jyn514 commented Aug 13, 2020

Uh oh!

jyn514 commented Aug 13, 2020

Uh oh!

ohaddahan commented Aug 14, 2020

Uh oh!

jyn514 commented Aug 15, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

ohaddahan commented Aug 11, 2020 •

edited by jyn514

Loading