Automate versioned Homebrew formula publishing, fix patch releases #864

courier-new · 2022-10-14T05:44:29Z

Closes #862.

Context

These GH action workflow changes enable us to automatically version Homebrew formulas for future src-cli releases while also preserving the "main" (unversioned) formula.

Before this, any new release would just overwrite the main formula. This means there was no way to get an older version of src-cli with brew unless someone manually published a versioned formula. But what's worse is this also means that if I wanted to cut a patch release for 3.43.3 with a backwards-compatible bug fix that was introduced after 4.0, and I used our standard release process, the next person who tried to install src-cli with brew install sourcegraph/src-cli/src-cli wouldn't get 4.0.2, they'd get 3.43.3. Not ideal.

Implementation overview

This is accomplished with 3 additional jobs to our GH action:

The first step, release_type determines what type of release we're doing: patch vs. update latest. This is because we need to make changes to the Homebrew tap repo ourselves if we're releasing a new latest version, and we need to use a different goreleaser config for either case due to Homebrew's naming convention expectations.

The second step, goreleaser_pre only runs if we're releasing a new latest version. It copies the current "main" formula in our Homebrew tap for the previous latest release to a versioned formula, so that it is preserved when goreleaser runs and overwrites the main formula for the latest build. It is not necessary to do this step if the latest version did not change.

Then the normal goreleaser step runs. If we're not releasing a new latest version, we use a different config which just publishes a versioned formula, ensuring we don't overwrite the main one. It also skips publishing the Docker image for src-cli:latest. If we are releasing the latest version, we just use the same config as before.

The third additional step, goreleaser_post also only runs if we're releasing a new latest version. It simply updates the name of the symlink to the main formula in the Homebrew tap repo. This enables someone to install the latest version either with or without the version. For example right now for 4.0.2:

# These will do the same thing
brew install sourcegraph/src-cli/src-cli
brew install sourcegraph/src-cli/src-cli@4.0.2

It is not necessary to update the symlink if the latest version did not change.

I added lots of comment inline to give clarity to what these steps are and why they are necessary. Ideally this would be something that goreleaser could handle for us automatically, but at the moment it's One Name Pattern to Rule Them All, and we'd rather not forego versions for a main formula, or vice versa.

I also updated our docs, as I realized we didn't even mention in the README that you can install src-cli with npm/npx.

Test plan

Obviously I haven't tested the full end-to-end workflow with a full release. It's possible something could still break.

Set up fork of Homebrew tap repo as if 4.0.1 was the latest release (main formula is for 4.0.1, versioned formulas up to 4.0.0, symlink for 4.0.1)
Update GH action to skip goreleaser and npm steps (don't want to actually publish anything new), point pre and post steps at fork Homebrew tap repo
Trigger action with fake github tag for an older patch release
Trigger action again with latest tag
Example GH actions workflow when releasing an older patch release version
- Step to copy main formula is skipped
- Goreleaser runs with patch config file
- Step to update symlink is skipped
Example GH actions workflow when releasing a new latest version
- Main formula is converted to versioned formula and result is committed (except not because --dry-run, but y'know)
- Goreleaser runs with normal config file
- Symlink is updated and result it committed (except not because --dry-run again)

courier-new · 2022-10-14T22:19:54Z

.github/workflows/goreleaser.yml

-    tags:
+    tags:	
      - '*'
-      - '!latest'


We don't use this tag anymore, so I deleted it so as not to be confusing.

courier-new · 2022-10-14T22:33:32Z

.github/workflows/goreleaser.yml

+          if [[ $(semver compare ${{ env.latest_tag }} ${{ env.most_recent_tag }}) == 0 ]]
+          then
+            echo "is_latest_version=1" >> $GITHUB_ENV
+          else
+            echo "is_latest_version=0" >> $GITHUB_ENV
+          fi


I couldn't figure out a way to write this as a one-liner but if someone who's better at bash-fu wants to give it a go, be my guest. 😄

I would prefer this over a one-liner. Easier to read.

courier-new · 2022-10-14T22:34:35Z

.github/workflows/goreleaser.yml

+      - name: Log variables
+        run: |
+          echo "Most recent (current) tag: ${{ env.most_recent_tag }}"
+          echo "Latest version tag: ${{ env.latest_tag }}"
+          echo "Building current version as new latest?: ${{ env.is_latest_version }}"


I could probably get rid of all this logging but it doesn't hurt to keep it, at least until we actually test an actual release. 🙂

courier-new · 2022-10-14T22:46:52Z

DEVELOPMENT.md

-For example, suppose we have the the recommended versions.
-
-| Sourcegraph version | Recommended src-cli version |
-| ------------------- | --------------------------- | 
-| `3.100`             | `3.90.5`                    |
-| `3.99`              | `3.85.7`                    |
-
-If a new feature is added to a new `3.91.6` release of src-cli and this change requires only features available in Sourcegraph `3.99`, then this feature should also be present in a new `3.85.8` release of src-cli. Because a Sourcegraph instance will automatically select the highest patch version, all non-breaking changes should increment only the patch version. 
-
-Note that if instead the recommended src-cli version for Sourcegraph `3.99` was `3.90.4` in the example above, there is no additional step required, and the new patch version of src-cli will be available to both Sourcegraph versions.


This example just confused me, since 99% of the time the recommended src-cli version matches the Sourcegraph version, so I decided to just remove it. Perhaps this was more common historically.

courier-new · 2022-10-14T22:51:21Z

.github/workflows/goreleaser.yml

+        # For latest and second latest tags, we can't use git tag --sort=version:refname
+        # because git doesn't have a concept of pre-release versions and thus mis-sorts
+        # versions like 4.0.0-rc.0 *after* 4.0.0.
+        run: |
+          echo "most_recent_tag=$(git tag --sort=taggerdate | tail -1)" >> $GITHUB_ENV
+          echo "latest_tag=$(git tag | tr - \~ | sort --version-sort | tr \~ - | tail -1)" >> $GITHUB_ENV
+          echo "second_latest_tag=$(git tag | tr - \~ | sort --version-sort | tr \~ - | tail -2 | sed -n 1p)" >> $GITHUB_ENV


As the comment above explains, git tag --sort=version:refname sorts by semantic version number but doesn't understand pre-release versions, so it sorts versions like 4.0.0-rc.0 after 4.0.0 when they should come before. I shamelessly stole this from a comment on this gist, but it worked reliably from my testing. To break it down, since I hate obtuse bash commands:

git tag lists all tags in src-cli.

tr - \~ replaces hyphens with tildes, so 4.0.0-rc.0 becomes 4.0.0~rc.0 (this is apparently what's typically understood as pre-releases by certain Linux distros)

sort --version-sort does natural sort by version number of the tags

tr \~ - replaces the tildes with hyphens again

tail -1 takes the last item, which based on the sorting should be the latest version tag

The command for the second-latest tag is the same, except it takes the second-to-last item instead.

Piszmog · 2022-10-17T18:37:35Z

.github/workflows/goreleaser.yml

+          if [[ $(semver compare ${{ env.latest_tag }} ${{ env.most_recent_tag }}) == 0 ]]
+          then
+            echo "is_latest_version=1" >> $GITHUB_ENV
+          else
+            echo "is_latest_version=0" >> $GITHUB_ENV
+          fi


I would prefer this over a one-liner. Easier to read.

)

courier-new added 6 commits October 13, 2022 12:17

WIP actions workflow

5fe29b5

Temporarily disable other actions for testing

1a43f06

WIP

94479f3

Cleanup

03f5628

Update README

88704af

WIP

8fb7009

courier-new self-assigned this Oct 14, 2022

Update README, add more comments

c858825

courier-new force-pushed the kr/automate-version-publishing-2 branch from 4f14561 to c858825 Compare October 14, 2022 20:08

courier-new added 4 commits October 14, 2022 13:26

Update release docs

a7dc824

Fix prerelease sorting

7bdcd35

Add comment about tag sorting

39c025c

Cleanup testing changes

57bc033

courier-new changed the base branch from kr/automate-version-publishing to main October 14, 2022 22:14

courier-new added 3 commits October 14, 2022 15:22

Use longer flag name to be clearer

320a82b

Remove extra space

578ff19

Dont publish src-cli:latest to Docker Hub for patch release config

72cf225

courier-new commented Oct 14, 2022

View reviewed changes

courier-new marked this pull request as ready for review October 14, 2022 22:47

courier-new requested a review from a team October 14, 2022 22:47

courier-new changed the title ~~[wip] automate versioned Homebrew formula publishing, fix patch releases~~ Automate versioned Homebrew formula publishing, fix patch releases Oct 14, 2022

courier-new commented Oct 14, 2022

View reviewed changes

courier-new mentioned this pull request Oct 14, 2022

[wip] automate versioned Homebrew formula publishing #863

Closed

Piszmog approved these changes Oct 17, 2022

View reviewed changes

BolajiOlajide approved these changes Oct 17, 2022

View reviewed changes

courier-new merged commit ecce113 into main Oct 18, 2022

courier-new deleted the kr/automate-version-publishing-2 branch October 18, 2022 00:11

scjohns pushed a commit that referenced this pull request Apr 24, 2023

Automate versioned Homebrew formula publishing, fix patch releases (#864

e779097

)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Automate versioned Homebrew formula publishing, fix patch releases #864

Automate versioned Homebrew formula publishing, fix patch releases #864

Uh oh!

courier-new commented Oct 14, 2022 •

edited

Loading

Uh oh!

courier-new Oct 14, 2022

Uh oh!

courier-new Oct 14, 2022

Uh oh!

Piszmog Oct 17, 2022

Uh oh!

courier-new Oct 14, 2022

Uh oh!

courier-new Oct 14, 2022

Uh oh!

courier-new Oct 14, 2022

Uh oh!

Piszmog Oct 17, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Automate versioned Homebrew formula publishing, fix patch releases #864

Automate versioned Homebrew formula publishing, fix patch releases #864

Uh oh!

Conversation

courier-new commented Oct 14, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Context

Implementation overview

Test plan

Uh oh!

courier-new Oct 14, 2022

Choose a reason for hiding this comment

Uh oh!

courier-new Oct 14, 2022

Choose a reason for hiding this comment

Uh oh!

Piszmog Oct 17, 2022

Choose a reason for hiding this comment

Uh oh!

courier-new Oct 14, 2022

Choose a reason for hiding this comment

Uh oh!

courier-new Oct 14, 2022

Choose a reason for hiding this comment

Uh oh!

courier-new Oct 14, 2022

Choose a reason for hiding this comment

Uh oh!

Piszmog Oct 17, 2022

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

courier-new commented Oct 14, 2022 •

edited

Loading