KAFKA-14995: Automate asf.yaml collaborators refresh by stevenbooke · Pull Request #13842 · apache/kafka

stevenbooke · 2023-06-12T00:09:20Z

Create Github Action workflow to run Python script which will automate the asf.yaml collaborators refresh.

Tested the workflow locally using https://github.com/nektos/act.

Committer Checklist (excluded from commit message)

[NA] Verify design and implementation
Verify test coverage and CI build status
Verify documentation (including upgrade notes)

stevenbooke · 2023-06-15T18:14:29Z

Could someone review this PR and leave some feedback please?

mimaison

Thanks for the PR. That seems a nice approach to automate this task. I left a few comments and suggestions.

mimaison · 2023-06-20T15:33:02Z

@@ -0,0 +1,24 @@
+name: Refresh asf.yaml collaborators every 3 months


We need to have the Apache license at the top of the file.

mimaison · 2023-06-20T15:33:05Z

@@ -0,0 +1,44 @@
+import os


We need to have the Apache license at the top of the file.

mimaison · 2023-06-20T15:43:40Z

+yaml_content = yaml.safe_load(file.decoded_content)
+
+# Update 'github_whitelist' list
+github_whitelist = refreshed_collaborators[:10]  # New users to be added


Isn't the length of refreshed_collaborators already 10? It's been assigned collaborators[:n] where n is 10.

Will fix that.

mimaison · 2023-06-20T15:44:14Z

+updated_yaml = yaml.safe_dump(yaml_content)
+
+# Commit and push the changes
+commit_message = "Update .asf.yaml file with refreshed github_whitelist, and collaborators"


We tend to prefix commit with either a Jira or MINOR. Maybe we can use MINOR: here

vvcephei

Hey @stevenbooke , thanks for this script!

I just have one concern (see below)

vvcephei · 2023-06-22T23:22:06Z

+### GET THE CONTRIBUTORS OF THE apache/kafka REPO ###
+n = 10
+repo = g.get_repo("apache/kafka")
+contributors = repo.get_contributors()


I'm worried about taking the top ten contributors by lifetime commits, rather than by commits from the last year (as in git shortlog --email --numbered --summary --since=2022-04-28), since we have some prolific contributors in the past who are no longer active.

Hello @vvcephei, I understand your concern. I will make changes to account for this.

…ntributors by lifetime commits, rather than by commits from the last year (as in git shortlog --email --numbered --summary --since=2022-04-28), since we have some prolific contributors in the past who are no longer active.'

mimaison · 2023-07-06T13:49:16Z

Thanks @stevenbooke for the updates. The changes seem fine to me. @vvcephei do you have further comments?

Also let's update apache/kafka-site#521 as we'll need to merge it first.

stevenbooke · 2023-07-17T19:13:52Z

@mimaison Do you think we can we move forward and merge this PR?

mimaison

Thanks for the updates, I left a couple of comments.

mimaison · 2023-07-18T09:35:31Z

+contributors_login_to_commit_volume = {}
+end_date = datetime.now()
+start_date = end_date - timedelta(days=365)
+for commit in repo.get_commits(since=start_date, until=end_date):


Here repo points to apache/kafka-site. Shouldn't it point to apache/kafka instead?

Yes, you are correct. Will change.

mimaison · 2023-07-18T09:36:43Z

+end_date = datetime.now()
+start_date = end_date - timedelta(days=365)
+for commit in repo.get_commits(since=start_date, until=end_date):
+    login = commit.author.login


author can be None is the author is not a member of Github, see PyGithub/PyGithub#279.

We should handle this case as this happens from time to time.

Thank you for pointing that out. I will make the appropriate changes.

mimaison · 2023-07-18T11:26:32Z

Also I inadvertently run the script and you can see the commit it generated: a1f6ab6

It's pretty different from the current file, so there are a few other issues. The jenkins section looks incorrect and it would be good to also keep the comments if possible.

mimaison · 2023-07-18T11:37:32Z

I also wonder if the tool should open a pull request instead of directly merging to trunk. It would make it easier to run the script locally as you can just close an unwanted PR while a commit as to be reverted.

stevenbooke · 2023-07-18T18:47:10Z

@mimaison As you have recommended, the script will create a new branch for the changes, commit the changes to the new branch, and open a pull request with the updated .asf.yaml file. The updated .asf.yaml file will be the same format as it was previously (it will retain the comments and only update the github_whitelist list and the collaborators list).

…an be 'None'. Ensure the updated '.asf.yaml' file retains the previous format ( retain the comments and only update the 'github_whitelist' list and the 'collaborators' list). Instead of directly merging to trunk, the script will create a new branch for the changes, commit the changes to the new branch, and open a pull request with the updated '.asf.yaml' file.

mimaison · 2023-07-19T10:05:53Z

+### GET THE NAMES OF THE KAFKA COMMITTERS FROM THE apache/kafka-site REPO ###
+github_token = os.environ.get('GITHUB_TOKEN')
+g = Github(github_token)
+repo = g.get_repo("apache/kafka-site")


Could we retrieve the organization from the environment? That would allow running this action on forks too.

We would not be able to retrieve the organization from the environment for "apache/kafka-site" due to the fact that "At the start of each workflow run, GitHub automatically creates a unique GITHUB_TOKEN secret to use in your workflow. You can use the GITHUB_TOKEN to authenticate in a workflow run.

When you enable GitHub Actions, GitHub installs a GitHub App on your repository. The GITHUB_TOKEN secret is a GitHub App installation access token. You can use the installation access token to authenticate on behalf of the GitHub App installed on your repository. The token's permissions are limited to the repository that contains your workflow." See reference here.

We would only be able to retrieve the organization from the environment for "apache/kafka".

mimaison · 2023-07-19T10:10:16Z

 start_date = end_date - timedelta(days=365)
+repo = g.get_repo("apache/kafka")
 for commit in repo.get_commits(since=start_date, until=end_date):
+    if commit.author is None and commit.author.login is None:


This should be or instead of and, both conditions will never be true together

Correct, will change.

mimaison · 2023-07-19T10:12:33Z


-# Commit and push the changes
+# Create a new branch for the changes
+new_branch_name = "update-asf.yaml-github-whitelist-and-collaborators"


Do you know whether/how these branches will be deleted if the PR is merged? Typically it's the author that can delete the branch. I'm wondering what happens the second time this runs, will it force push to the branch or fail?

I look into this more in the coming days and post an update.

@mimaison If the PR is merged the branch will not be deleted.

I have updated the script to check if branch already exists. If so, the script will commit the changes to the branch, and open a pull request with the updated .asf.yaml file. Otherwise, the script will create a new branch for the changes, commit the changes to the new branch, and open a pull request with the updated .asf.yaml file.

Here is an example of what happens if changes are made to a branch that already has a pull request opened and merged:

A pull request is opened for the branch.
The pull request is merged.
Changes are made to the branch.
Another pull request is opened for the branch.
The second pull request will be based on the latest commit on the branch, which will include all of the changes that have been made to the branch since the first pull request was merged.

…nges to the branch, and open a pull request with the updated '.asf.yaml' file. Otherwise, the script will create a new branch for the changes, commit the changes to the new branch, and open a pull request with the updated '.asf.yaml' file.

mimaison · 2023-09-11T15:04:36Z

@stevenbooke Thanks for the updates and sorry for the delay. I took another look at this PR today.
When trying to run the script, I get the following error:

Traceback (most recent call last):
  File "/private/tmp/test/./refresh-collaborators.py", line 79, in <module>
    branch = repo.get_branch(branch=branch_name)
  File "/private/tmp/test/lib/python3.9/site-packages/github/Repository.py", line 1939, in get_branch
    headers, data = self._requester.requestJsonAndCheck(
  File "/private/tmp/test/lib/python3.9/site-packages/github/Requester.py", line 442, in requestJsonAndCheck
    return self.__check(
  File "/private/tmp/test/lib/python3.9/site-packages/github/Requester.py", line 487, in __check
    raise self.__createException(status, responseHeaders, data)
github.GithubException.GithubException: 404 {"message": "Branch not found", "documentation_url": "https://docs.github.com/rest/branches/branches#get-a-branch"}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/private/tmp/test/./refresh-collaborators.py", line 90, in <module>
    repo.create_git_ref(f"refs/heads/{branch_name}", repo.default_branch)
  File "/private/tmp/test/lib/python3.9/site-packages/github/Repository.py", line 1085, in create_git_ref
    headers, data = self._requester.requestJsonAndCheck(
  File "/private/tmp/test/lib/python3.9/site-packages/github/Requester.py", line 442, in requestJsonAndCheck
    return self.__check(
  File "/private/tmp/test/lib/python3.9/site-packages/github/Requester.py", line 487, in __check
    raise self.__createException(status, responseHeaders, data)
github.GithubException.GithubException: 422 {"message": "Invalid request.\n\nAt least 40 characters are required; only 5 were supplied.", "documentation_url": "https://docs.github.com/rest/git/refs#create-a-reference"}

Is this expected?

github-actions · 2023-12-11T03:33:31Z

This PR is being marked as stale since it has not had any activity in 90 days. If you would like to keep this PR alive, please ask a committer for review. If the PR has merge conflicts, please update it with the latest from trunk (or appropriate release branch)

If this PR is no longer valid or desired, please feel free to close it. If no activity occurs in the next 30 days, it will be automatically closed.

mimaison · 2024-01-11T10:41:35Z

@stevenbooke Do you intend to finish this PR?

fonsdant · 2024-09-07T13:39:57Z

@mimaison, @stevenbooke could I try to resume this work?

Automate asf.yaml collaborators refresh

94d9cf9

mimaison reviewed Jun 20, 2023

View reviewed changes

mimaison mentioned this pull request Jun 20, 2023

KAFKA-14995: Automate asf.yaml collaborators refresh apache/kafka-site#521

Merged

vvcephei reviewed Jun 22, 2023

View reviewed changes

mimaison reviewed Jul 18, 2023

View reviewed changes

mimaison mentioned this pull request Jul 18, 2023

Revert "MINOR: Update .asf.yaml file with refreshed github_whitelist,… #14037

Merged

3 tasks

mimaison reviewed Jul 19, 2023

View reviewed changes

github-actions Bot added the stale Stale PRs label Dec 11, 2023

stevenbooke closed this by deleting the head repository May 3, 2024

fonsdant mentioned this pull request Sep 8, 2024

KAFKA-14995: Automate asf.yaml collaborators refresh #17124

Merged

		@@ -0,0 +1,24 @@
		name: Refresh asf.yaml collaborators every 3 months

Conversation

stevenbooke commented Jun 12, 2023

Committer Checklist (excluded from commit message)

Uh oh!

stevenbooke commented Jun 15, 2023

Uh oh!

mimaison left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vvcephei left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mimaison commented Jul 6, 2023

Uh oh!

stevenbooke commented Jul 17, 2023

Uh oh!

mimaison left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mimaison commented Jul 18, 2023

Uh oh!

mimaison commented Jul 18, 2023

Uh oh!

stevenbooke commented Jul 18, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mimaison commented Sep 11, 2023

Uh oh!

github-actions Bot commented Dec 11, 2023

Uh oh!

mimaison commented Jan 11, 2024

Uh oh!

fonsdant commented Sep 7, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

stevenbooke commented Jul 18, 2023 •

edited

Loading