From 66a0bfd5241d136e6cc4c7602b53310fc7fbd074 Mon Sep 17 00:00:00 2001 From: comphead Date: Fri, 17 Apr 2026 09:02:51 -0700 Subject: [PATCH 1/5] chore: Update Release instructions --- dev/release/README.md | 147 +++++++++++++++++++++--------------------- 1 file changed, 74 insertions(+), 73 deletions(-) diff --git a/dev/release/README.md b/dev/release/README.md index 2f99d52b200d2..f619fdb9b1933 100644 --- a/dev/release/README.md +++ b/dev/release/README.md @@ -77,7 +77,7 @@ We then publish the code in the approved artifacts to crates.io. ### 1. Create Release Branch -First create a new release branch from `main` in the apache repository. +First create a new release branch from `main` in the `apache` repository. For example, to create the `branch-50` branch for the `50.x.y` release series: @@ -88,11 +88,28 @@ git checkout -b branch-50 # create local branch git push -u apache branch-50 # push branch to apache remote ``` -### 2. Add a protection to release candidate branch +### 2. Prepare PR to Update the Release Version -To protect a release candidate branch from accidental merges, run: +Manually update the DataFusion version in the root `Cargo.toml` to +reflect the new release version. Ensure `Cargo.lock` is updated accordingly by +running: + +```shell +cargo check -p datafusion +``` + +Then commit the changes and create a PR targeting the release branch. ```shell +git commit -a -m 'Update version' +``` + +### 3. Sync Update Release Version to Main and Branch Protection + +To protect a release candidate branch from accidental merges, create another PR against `main``: + +```shell +git fetch upstream && git checkout -b sync_branch ./dev/release/add-branch-protection.sh 50 ``` @@ -104,28 +121,22 @@ branch-50: required_approving_review_count: 1 ``` -- Create a PR. +- Cherry-pick the update version changes from step 2 +- Commit changes +- Push to `origin/sync_branch` +- Create a PR against `main`. - Merge to `main`. -### 3. Prepare PR to Update Changelog and the Release Version +### 4. Backporting urgent changes -First, prepare a PR to update the changelog and versions to reflect the planned -release. See [#18173](https://github.com/apache/datafusion/pull/18173) for an example. +After release branch `branch-N` created, got its version updated and protected, please check if there are any backports expected from the community. +Please refer to [Backport Flow](../../docs/source/contributor-guide/release_management.md#backport-workflow) for more details. -- This PR should be the last one merged before creating a release - candidate, so the changelog includes all changes. -- If there are code changes between RCs, create and merge a new PR to update - the changelog before creating the next RC. +Backports are important and sometimes unexpected, so please proceed to next release steps once all expected backports are applied. -**Step 1**: Manually update the DataFusion version in the root `Cargo.toml` to -reflect the new release version. Ensure `Cargo.lock` is updated accordingly by -running: +### 5. Prepare PR to Update Changelog -```shell -cargo check -p datafusion -``` - -**Step 2**: Update the [changelog] in `dev/changelog/`. Each release has its +Update the [changelog] in `dev/changelog/`. Each release has its own file, such as `dev/changelog/50.0.0.md`, which should include all changes since the previous release. @@ -168,14 +179,25 @@ prettier -w dev/changelog/51.0.0.md Then commit the changes and create a PR targeting the release branch. ```shell -git commit -a -m 'Update version' +git commit -a -m 'Update changelog' ``` -Remember to merge any fixes back to the `main` branch as well. +### 6. Sync Changelog to main +To sync changelog create another PR against `main``: -### 4. Prepare Release Candidate Artifacts +```shell +git fetch upstream && git checkout -b sync_change_log +``` -After the PR gets merged, you are ready to create release artifacts based off the +- Cherry-pick the changelog updates from step 5 +- Commit changes +- Push to `origin/sync_change_log` +- Create a PR against `main`. +- Merge to `main`. + +### 7. Prepare Release Candidate Artifacts + +After the changelog updates merged to `main`, you are ready to create release artifacts based off the merged commit. - You must be a committer to run these scripts because they upload to the @@ -191,20 +213,12 @@ Pick numbers in sequential order, with `1` for `rc1`, `2` for `rc2`, etc. While the official release artifacts are signed tarballs and zip files, we also tag the commit it was created from for convenience and code archaeology. Release tags -look like `38.0.0`, and release candidate tags look like `38.0.0-rc0`. See [the list of existing +look like `50.3.0`, and release candidate tags look like `50.3.0-rc1`. See [the list of existing tags]. [the list of existing tags]: https://github.com/apache/datafusion/tags -Create and push the RC tag: - -```shell -git fetch apache -git tag -rc apache/branch-X -git push apache -rc -``` - -For example, to create the `50.3.0-rc1` tag from `branch-50`, use: +Create and push the RC tag, for example, to create the `50.3.0-rc1` tag from `branch-50`, use: ```shell git fetch apache @@ -212,6 +226,8 @@ git tag 50.3.0-rc1 apache/branch-50 git push apache 50.3.0-rc1 ``` +Please make sure the format is correct, tools like Homebrew listens for tags and in case of malformed tags users would be notified for non-existent version + #### Create, Sign, and Upload Artifacts Run the `create-tarball.sh` script with the `` tag and `` number you determined in previous steps: @@ -226,12 +242,12 @@ The `create-tarball.sh` script 1. Creates and uploads all release candidate artifacts to the [datafusion dev](https://dist.apache.org/repos/dist/dev/datafusion) location on the - apache distribution SVN server + Apache distribution SVN server 2. Provides you an email template to send to `dev@datafusion.apache.org` for release voting. -### 5. Vote on Release Candidate Artifacts +### 8. Vote on Release Candidate Artifacts Send the email output from the script to `dev@datafusion.apache.org`. @@ -248,15 +264,10 @@ review the release candidate. ./dev/release/verify-release-candidate.sh 50.3.0 1 ``` -#### If the Release is not Approved - -If the release is not approved, fix whatever the problem is, make a new release -candidate and try again. +#### If Changes Are Requested -Reminders: +If the release is not approved or urgent backports requested, please start over from [here](#4-backporting-urgent-changes) -1. Update the changelog and create a new PR if there are any code changes, and merge it before creating the next RC. -2. Merge any fixes back to the `main` branch as well. #### If the Vote Passes: Announce the Result @@ -273,9 +284,7 @@ with the release verification. ### 6. Finalize the Release -NOTE: steps in this section can only be done by PMC members. - -#### After the release is approved +NOTE: steps in this section can only be done by PMC members after release is approved. Move artifacts to the release location in SVN, e.g. https://dist.apache.org/repos/dist/release/datafusion/datafusion-50.3.0/, using @@ -292,7 +301,7 @@ Congratulations! The release is now official! Tag the same release candidate commit with the final release tag ```shell -git co apache/50.3.0-rc1 +git checkout 50.3.0-rc1 git tag 50.3.0 git push apache 50.3.0 ``` @@ -314,48 +323,48 @@ to all DataFusion crates. Download and unpack the official release tarball Verify that the Cargo.toml in the tarball contains the correct version -(e.g. `version = "38.0.0"`) and then publish the crates by running the following commands +(e.g. `version = "50.3.0"`) and then publish the crates by running the following commands ```shell -(cd datafusion/doc && cargo publish) -(cd datafusion/common-runtime && cargo publish) (cd datafusion/common && cargo publish) -(cd datafusion/proto-common && cargo publish) -(cd datafusion/macros && cargo publish) +(cd datafusion/common-runtime && cargo publish) +(cd datafusion/doc && cargo publish) (cd datafusion/expr-common && cargo publish) +(cd datafusion/macros && cargo publish) +(cd datafusion/proto-common && cargo publish) (cd datafusion/physical-expr-common && cargo publish) -(cd datafusion/functions-window-common && cargo publish) (cd datafusion/functions-aggregate-common && cargo publish) +(cd datafusion/functions-window-common && cargo publish) (cd datafusion/expr && cargo publish) (cd datafusion/execution && cargo publish) -(cd datafusion/physical-expr && cargo publish) (cd datafusion/functions && cargo publish) -(cd datafusion/functions-window && cargo publish) +(cd datafusion/physical-expr && cargo publish) (cd datafusion/functions-aggregate && cargo publish) +(cd datafusion/functions-window && cargo publish) (cd datafusion/physical-expr-adapter && cargo publish) -(cd datafusion/physical-plan && cargo publish) (cd datafusion/functions-nested && cargo publish) +(cd datafusion/physical-plan && cargo publish) (cd datafusion/session && cargo publish) -(cd datafusion/datasource && cargo publish) -(cd datafusion/catalog && cargo publish) -(cd datafusion/spark && cargo publish) (cd datafusion/sql && cargo publish) +(cd datafusion/datasource && cargo publish) (cd datafusion/optimizer && cargo publish) -(cd datafusion/functions-table && cargo publish) -(cd datafusion/datasource-json && cargo publish) -(cd datafusion/datasource-csv && cargo publish) -(cd datafusion/datasource-avro && cargo publish) +(cd datafusion/catalog && cargo publish) (cd datafusion/datasource-arrow && cargo publish) +(cd datafusion/datasource-avro && cargo publish) +(cd datafusion/datasource-csv && cargo publish) +(cd datafusion/datasource-json && cargo publish) (cd datafusion/pruning && cargo publish) (cd datafusion/datasource-parquet && cargo publish) -(cd datafusion/catalog-listing && cargo publish) +(cd datafusion/functions-table && cargo publish) (cd datafusion/physical-optimizer && cargo publish) +(cd datafusion/catalog-listing && cargo publish) (cd datafusion/core && cargo publish) (cd datafusion-cli && cargo publish) -(cd datafusion/substrait && cargo publish) -(cd datafusion/sqllogictest && cargo publish) (cd datafusion/proto && cargo publish) +(cd datafusion/spark && cargo publish) +(cd datafusion/substrait && cargo publish) (cd datafusion/ffi && cargo publish) +(cd datafusion/sqllogictest && cargo publish) ``` ### Publish datafusion-cli on Homebrew @@ -409,11 +418,3 @@ To delete a release: ```shell svn delete -m "delete old DataFusion release" https://dist.apache.org/repos/dist/release/datafusion/datafusion-50.0.0 ``` - -### 11. Forward-port changelog to `main` - -After the release is published, forward port the version update and changelog to -`main` so that it is up to date for the next release. See [#21053] for an -example PR that forward-ports the changelog to `main`. - -[#21053]: https://github.com/apache/datafusion/pull/21053 From 923ecd12de56cb239a0514889b3312bd6364244e Mon Sep 17 00:00:00 2001 From: comphead Date: Fri, 17 Apr 2026 09:11:32 -0700 Subject: [PATCH 2/5] chore: Update Release instructions --- dev/release/README.md | 18 ++++++++++++++++-- 1 file changed, 16 insertions(+), 2 deletions(-) diff --git a/dev/release/README.md b/dev/release/README.md index f619fdb9b1933..020761887a5c0 100644 --- a/dev/release/README.md +++ b/dev/release/README.md @@ -183,6 +183,7 @@ git commit -a -m 'Update changelog' ``` ### 6. Sync Changelog to main + To sync changelog create another PR against `main``: ```shell @@ -226,7 +227,7 @@ git tag 50.3.0-rc1 apache/branch-50 git push apache 50.3.0-rc1 ``` -Please make sure the format is correct, tools like Homebrew listens for tags and in case of malformed tags users would be notified for non-existent version +Please make sure the format is correct, tools like Homebrew listens for tags and in case of malformed tags users would be notified for non-existent version #### Create, Sign, and Upload Artifacts @@ -268,7 +269,6 @@ review the release candidate. If the release is not approved or urgent backports requested, please start over from [here](#4-backporting-urgent-changes) - #### If the Vote Passes: Announce the Result Call the vote on the Arrow dev list by replying to the RC voting thread. The @@ -367,6 +367,20 @@ Verify that the Cargo.toml in the tarball contains the correct version (cd datafusion/sqllogictest && cargo publish) ``` +Crates.io publishing depends on crates dependency tree, this list might contain wrong order. +If it happens crates.io fails with wrong dependency message like below, just rerun all publishing commands. + +```shell +error: failed to prepare local package for uploading + +Caused by: + failed to select a version for the requirement `datafusion-proto = "^53.1.0"` + candidate versions found which didn't match: 53.0.0, 52.5.0, 52.4.0, ... + location searched: crates.io index + required by package `datafusion-ffi v53.1.0 (/private/tmp/apache-datafusion-53.1.0/datafusion/ffi)` +MacBook-Pro-135:apache-datafusion-53.1.0 ovoievodin$ (cd datafusion/proto && cargo publish) +``` + ### Publish datafusion-cli on Homebrew Note: [`datafusion` formula](https://formulae.brew.sh/formula/datafusion) is [updated automatically](https://github.com/Homebrew/homebrew-core/pulls?q=is%3Apr+datafusion+is%3Aclosed), From 8cc72eafc6a0b0b07db0e0db9661bb4465489322 Mon Sep 17 00:00:00 2001 From: comphead Date: Fri, 17 Apr 2026 12:24:30 -0700 Subject: [PATCH 3/5] chore: Update Release instructions --- dev/release/README.md | 55 ++++++++++++++++++++++--------------------- 1 file changed, 28 insertions(+), 27 deletions(-) diff --git a/dev/release/README.md b/dev/release/README.md index 020761887a5c0..1cc0655f10e49 100644 --- a/dev/release/README.md +++ b/dev/release/README.md @@ -98,18 +98,18 @@ running: cargo check -p datafusion ``` -Then commit the changes and create a PR targeting the release branch. +Then commit the changes and create a PR targeting the release branch `branch-N`. ```shell git commit -a -m 'Update version' ``` -### 3. Sync Update Release Version to Main and Branch Protection +### 3. Protect Release Branch -To protect a release candidate branch from accidental merges, create another PR against `main``: +To protect a release candidate branch from accidental merges, create PR against `main`: ```shell -git fetch upstream && git checkout -b sync_branch +git fetch upstream && git checkout -b protect_branch_50 ./dev/release/add-branch-protection.sh 50 ``` @@ -121,15 +121,15 @@ branch-50: required_approving_review_count: 1 ``` -- Cherry-pick the update version changes from step 2 - Commit changes -- Push to `origin/sync_branch` +- Push to `origin/protect_branch_50` - Create a PR against `main`. - Merge to `main`. +- Notify community in Discord/Slack that release branch is created ### 4. Backporting urgent changes -After release branch `branch-N` created, got its version updated and protected, please check if there are any backports expected from the community. +After release branch `branch-N` created, protected and got its version updated, please check if there are any backports expected from the community. Please refer to [Backport Flow](../../docs/source/contributor-guide/release_management.md#backport-workflow) for more details. Backports are important and sometimes unexpected, so please proceed to next release steps once all expected backports are applied. @@ -182,21 +182,7 @@ Then commit the changes and create a PR targeting the release branch. git commit -a -m 'Update changelog' ``` -### 6. Sync Changelog to main - -To sync changelog create another PR against `main``: - -```shell -git fetch upstream && git checkout -b sync_change_log -``` - -- Cherry-pick the changelog updates from step 5 -- Commit changes -- Push to `origin/sync_change_log` -- Create a PR against `main`. -- Merge to `main`. - -### 7. Prepare Release Candidate Artifacts +### 6. Prepare Release Candidate Artifacts After the changelog updates merged to `main`, you are ready to create release artifacts based off the merged commit. @@ -282,7 +268,7 @@ The vote has passed with +1 votes. Thank you to all who helped with the release verification. ``` -### 6. Finalize the Release +### 7. Finalize the Release NOTE: steps in this section can only be done by PMC members after release is approved. @@ -296,7 +282,7 @@ the `release-tarball.sh` script: Congratulations! The release is now official! -### 7. Create Release git tags +### 8. Create Release git tags Tag the same release candidate commit with the final release tag @@ -306,7 +292,7 @@ git tag 50.3.0 git push apache 50.3.0 ``` -### 8. Publish on Crates.io +### 9. Publish on Crates.io Only approved releases of the tarball should be published to crates.io, in order to conform to Apache Software Foundation @@ -386,7 +372,22 @@ MacBook-Pro-135:apache-datafusion-53.1.0 ovoievodin$ (cd datafusion/proto && car Note: [`datafusion` formula](https://formulae.brew.sh/formula/datafusion) is [updated automatically](https://github.com/Homebrew/homebrew-core/pulls?q=is%3Apr+datafusion+is%3Aclosed), so no action is needed. -### 9: Add the release to Apache Reporter +### 10. Sync Changelog and Version to main + +To sync changelog version and create PR against `main`: + +```shell +git fetch upstream && git checkout -b sync_change_log_version +``` + +- Cherry-pick or patch the version updates from step 2 +- Cherry-pick or patch the changelog updates from step 5 +- Commit changes, `git commit -a -m 'Sync changelog and version to main'` +- Push to `origin/sync_change_log_version` +- Create a PR against `main`. +- Merge to `main`. + +### 10: Add the release to Apache Reporter When you have published the release, please help the project by adding the release to [Apache Reporter](https://reporter.apache.org/). The reporter system should @@ -397,7 +398,7 @@ the examples from previous releases. The release information is used to generate a template for a board report (see example from Apache Arrow project [here](https://github.com/apache/arrow/pull/14357)). -### 10: Delete Old RCs and Releases +### 11: Delete Old RCs and Releases See the ASF documentation on [when to archive](https://www.apache.org/legal/release-policy.html#when-to-archive) for more information. From fcbf5df74bbd55038662f90459dfc69d44ebc1c1 Mon Sep 17 00:00:00 2001 From: comphead Date: Fri, 17 Apr 2026 12:37:27 -0700 Subject: [PATCH 4/5] chore: Update Release instructions --- dev/release/README.md | 26 +++++++++++++------------- dev/release/download-python-wheels.py | 4 ++-- 2 files changed, 15 insertions(+), 15 deletions(-) diff --git a/dev/release/README.md b/dev/release/README.md index 1cc0655f10e49..9642f2f6af745 100644 --- a/dev/release/README.md +++ b/dev/release/README.md @@ -109,7 +109,7 @@ git commit -a -m 'Update version' To protect a release candidate branch from accidental merges, create PR against `main`: ```shell -git fetch upstream && git checkout -b protect_branch_50 +git fetch apache && git checkout -b protect_branch_50 ./dev/release/add-branch-protection.sh 50 ``` @@ -136,7 +136,7 @@ Backports are important and sometimes unexpected, so please proceed to next rele ### 5. Prepare PR to Update Changelog -Update the [changelog] in `dev/changelog/`. Each release has its +Update the changelog in `dev/changelog/`. Each release has its own file, such as `dev/changelog/50.0.0.md`, which should include all changes since the previous release. @@ -184,7 +184,7 @@ git commit -a -m 'Update changelog' ### 6. Prepare Release Candidate Artifacts -After the changelog updates merged to `main`, you are ready to create release artifacts based off the +After the changelog updates merged to `branch-N`, you are ready to create release artifacts based off the merged commit. - You must be a committer to run these scripts because they upload to the @@ -222,7 +222,7 @@ Run the `create-tarball.sh` script with the `` tag and `` number yo For example, to create the `50.3.0-rc1` artifacts: ```shell -GH_TOKEN= ./dev/release/create-tarball.sh 50.3.0 1 +GITHUB_TOKEN= ./dev/release/create-tarball.sh 50.3.0 1 ``` The `create-tarball.sh` script @@ -234,7 +234,7 @@ The `create-tarball.sh` script 2. Provides you an email template to send to `dev@datafusion.apache.org` for release voting. -### 8. Vote on Release Candidate Artifacts +### 7. Vote on Release Candidate Artifacts Send the email output from the script to `dev@datafusion.apache.org`. @@ -257,7 +257,7 @@ If the release is not approved or urgent backports requested, please start over #### If the Vote Passes: Announce the Result -Call the vote on the Arrow dev list by replying to the RC voting thread. The +Call the vote on the Apache Datafusion dev list by replying to the RC voting thread. The reply should have a new subject constructed by adding the `[RESULT]` prefix to the old subject line. @@ -268,7 +268,7 @@ The vote has passed with +1 votes. Thank you to all who helped with the release verification. ``` -### 7. Finalize the Release +### 8. Finalize the Release NOTE: steps in this section can only be done by PMC members after release is approved. @@ -282,7 +282,7 @@ the `release-tarball.sh` script: Congratulations! The release is now official! -### 8. Create Release git tags +### 9. Create Release git tags Tag the same release candidate commit with the final release tag @@ -292,7 +292,7 @@ git tag 50.3.0 git push apache 50.3.0 ``` -### 9. Publish on Crates.io +### 10. Publish on Crates.io Only approved releases of the tarball should be published to crates.io, in order to conform to Apache Software Foundation @@ -372,12 +372,12 @@ MacBook-Pro-135:apache-datafusion-53.1.0 ovoievodin$ (cd datafusion/proto && car Note: [`datafusion` formula](https://formulae.brew.sh/formula/datafusion) is [updated automatically](https://github.com/Homebrew/homebrew-core/pulls?q=is%3Apr+datafusion+is%3Aclosed), so no action is needed. -### 10. Sync Changelog and Version to main +### 11. Sync Changelog and Version to main To sync changelog version and create PR against `main`: ```shell -git fetch upstream && git checkout -b sync_change_log_version +git fetch apache && git checkout -b sync_change_log_version ``` - Cherry-pick or patch the version updates from step 2 @@ -387,7 +387,7 @@ git fetch upstream && git checkout -b sync_change_log_version - Create a PR against `main`. - Merge to `main`. -### 10: Add the release to Apache Reporter +### 12. Add the release to Apache Reporter When you have published the release, please help the project by adding the release to [Apache Reporter](https://reporter.apache.org/). The reporter system should @@ -398,7 +398,7 @@ the examples from previous releases. The release information is used to generate a template for a board report (see example from Apache Arrow project [here](https://github.com/apache/arrow/pull/14357)). -### 11: Delete Old RCs and Releases +### 13. Delete Old RCs and Releases See the ASF documentation on [when to archive](https://www.apache.org/legal/release-policy.html#when-to-archive) for more information. diff --git a/dev/release/download-python-wheels.py b/dev/release/download-python-wheels.py index 3f97d0a05c3d0..b0998d4dc7733 100644 --- a/dev/release/download-python-wheels.py +++ b/dev/release/download-python-wheels.py @@ -39,11 +39,11 @@ def main(): args = parser.parse_args() tag = args.tag - ghp_token = os.environ.get("GH_TOKEN") + ghp_token = os.environ.get("GITHUB_TOKEN") if not ghp_token: print( "ERROR: Personal Github token is required to download workflow artifacts. " - "Please specify a token through GH_TOKEN environment variable.") + "Please specify a token through GITHUB_TOKEN environment variable.") sys.exit(1) print(f"Downloading latest python wheels for RC tag {tag}...") From 60dcbce7181b5bef27dd8d488893451b24b4eb9d Mon Sep 17 00:00:00 2001 From: comphead Date: Mon, 20 Apr 2026 10:00:27 -0700 Subject: [PATCH 5/5] chore: Update Release instructions --- dev/release/README.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/dev/release/README.md b/dev/release/README.md index 9642f2f6af745..2ca495cbb135f 100644 --- a/dev/release/README.md +++ b/dev/release/README.md @@ -257,7 +257,7 @@ If the release is not approved or urgent backports requested, please start over #### If the Vote Passes: Announce the Result -Call the vote on the Apache Datafusion dev list by replying to the RC voting thread. The +Call the vote on the Datafusion dev list by replying to the RC voting thread. The reply should have a new subject constructed by adding the `[RESULT]` prefix to the old subject line. @@ -364,7 +364,6 @@ Caused by: candidate versions found which didn't match: 53.0.0, 52.5.0, 52.4.0, ... location searched: crates.io index required by package `datafusion-ffi v53.1.0 (/private/tmp/apache-datafusion-53.1.0/datafusion/ffi)` -MacBook-Pro-135:apache-datafusion-53.1.0 ovoievodin$ (cd datafusion/proto && cargo publish) ``` ### Publish datafusion-cli on Homebrew