From 82474276ca9072cb761d26860eef810f3254053a Mon Sep 17 00:00:00 2001 From: Otto van der Schaaf Date: Sun, 15 Dec 2019 15:28:14 +0100 Subject: [PATCH 1/6] Add RELEASE_PROCEDURE.md Signed-off-by: Otto van der Schaaf --- RELEASE_PROCEDURE.md | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) create mode 100644 RELEASE_PROCEDURE.md diff --git a/RELEASE_PROCEDURE.md b/RELEASE_PROCEDURE.md new file mode 100644 index 000000000..af616661e --- /dev/null +++ b/RELEASE_PROCEDURE.md @@ -0,0 +1,17 @@ +# Release procedure + +- Update [version_history.md](docs/root/version_history.md). + - Make sure breaking changes are explicitly called out. + Ideally these have been tracked on the go but it does not hurt to ask around and double + check. Running the python integration tests of the previous release with the latest binaries could serve as a double check for this. + - Ensure the release notes are complete. Ideally these have been tracked on the go. +- Perform some thorough testing to double down on stability. + +```bash +bazel test --cache_test_results=no --test_env=ENVOY_IP_TEST_VERSIONS=all --runs_per_test=1000 --jobs 50 -c dbg --local_resources 20000,20,0.25 //test:* +``` + +- Draft a [GitHub tagged release](https://github.com/envoyproxy/nighthawk/releases/new) using the [current version number on master]((include/nighthawk/common/version.h)). Earlier releases are tagged like `v0.1` so currently we use that as a naming convention. +- [File an issue](https://github.com/envoyproxy/nighthawk/issues/new?title=[VOTE]+Release+v0.x&body=Release%20X%20is%20ready%20for%20review.%20Please%20take%20a%20look%20and%20vote!) to raise a vote on publishing the drafted release and point contributors to that. +- If nobody raises blocking issues, it's time to go ahead and publish the drafted release! +- Bump `MAJOR_VERSION` / `MINOR_VERSION` in [version.h](include/nighthawk/common/version.h) to the next version. From 2aee23e1f8ff2aa9f7b5749858beba3459334e2e Mon Sep 17 00:00:00 2001 From: Otto van der Schaaf Date: Wed, 18 Dec 2019 11:18:20 +0100 Subject: [PATCH 2/6] iteration on the release procedure Signed-off-by: Otto van der Schaaf --- RELEASE_PROCEDURE.md | 36 +++++++++++++++++++++++++++++------- 1 file changed, 29 insertions(+), 7 deletions(-) diff --git a/RELEASE_PROCEDURE.md b/RELEASE_PROCEDURE.md index af616661e..107682f7e 100644 --- a/RELEASE_PROCEDURE.md +++ b/RELEASE_PROCEDURE.md @@ -1,17 +1,39 @@ # Release procedure -- Update [version_history.md](docs/root/version_history.md). +## Preparation + +- Consolidate the [release notes](docs/root/version_history.md). - Make sure breaking changes are explicitly called out. Ideally these have been tracked on the go but it does not hurt to ask around and double check. Running the python integration tests of the previous release with the latest binaries could serve as a double check for this. - Ensure the release notes are complete. Ideally these have been tracked on the go. -- Perform some thorough testing to double down on stability. +- Based on our [semantic versioning](https://semver.org/spec/v2.0.0.html) strategy, our release numbers look like `MAJOR.MINOR.PATCH`. Determine the release type to come up with what the next version should look like: + - Breaking changes increment `MAJOR`, and reset `MINOR.PATCH` to `0.0` + - Added functionality which is backwards compatible increments `MINOR`, and resets `PATCH` to `0` + - Bug fixes increment `PATCH` + +## Release steps + +1. Speculatively bump the version in [version.h](include/nighthawk/common/version.h) to the version you determined earlier. This may result in version gaps if a release attempt fails, but avoids having to freeze merges to master and/or having to work with release branches. In short it helps keeping the release procedure lean and mean and eliminates the need for blocking others while this procedure is in-flight. +2. Draft a [GitHub tagged release](https://github.com/envoyproxy/nighthawk/releases/new). Earlier releases are tagged like `v0.1`, but as of `0.3.0`we are using [semantic versioning](https://semver.org/spec/v2.0.0.html) +3. Perform thorough testing of the targeted revision to double down on stability [1] +4. Create an optimized build for comparing with the previous release. Changes in performance + and/or measurement accuracy may need to be considered breaking. If in doubt, discuss! +5. If things look good, [File an issue](https://github.com/envoyproxy/nighthawk/issues/new?title=[VOTE]+Release+v0.x&body=Release%20X%20is%20ready%20for%20review.%20Please%20take%20a%20look%20and%20vote!) to raise a vote on publishing the drafted release, and point contributors to that. +6. Allow sufficient time to transpire for others to provide feedback, before moving on to the next step. If nobody raises blocking issues, it is time to go ahead and publish the drafted release. +7. |o/ Announce the new release on Slack, Twitter + +In case any of the steps above prevent shipping: + +1. Discard the draft release tag, if any +2. Close the voting issue, if any. If needed share context in the issue so everyone has context. +3. Subsequently re-spin the full procedure from the top after the blocking issue is resolved + +[1] Consider running the unit tests and integration tests repeatedly and concurrently for while. +It's worth noting that some of the integration tests have not been designed for this, so that part +is not tried and proven yet. (some of the integration tests are sensitive to timing, and may get +unstable when slowed down due to concurrent runs). A sample command to do this: ```bash bazel test --cache_test_results=no --test_env=ENVOY_IP_TEST_VERSIONS=all --runs_per_test=1000 --jobs 50 -c dbg --local_resources 20000,20,0.25 //test:* ``` - -- Draft a [GitHub tagged release](https://github.com/envoyproxy/nighthawk/releases/new) using the [current version number on master]((include/nighthawk/common/version.h)). Earlier releases are tagged like `v0.1` so currently we use that as a naming convention. -- [File an issue](https://github.com/envoyproxy/nighthawk/issues/new?title=[VOTE]+Release+v0.x&body=Release%20X%20is%20ready%20for%20review.%20Please%20take%20a%20look%20and%20vote!) to raise a vote on publishing the drafted release and point contributors to that. -- If nobody raises blocking issues, it's time to go ahead and publish the drafted release! -- Bump `MAJOR_VERSION` / `MINOR_VERSION` in [version.h](include/nighthawk/common/version.h) to the next version. From 696410260a3dcfab4d9b48823fcfcff6ddc37b00 Mon Sep 17 00:00:00 2001 From: Otto van der Schaaf Date: Fri, 20 Dec 2019 10:25:00 +0100 Subject: [PATCH 3/6] Review: address feedback Signed-off-by: Otto van der Schaaf --- RELEASE_PROCEDURE.md | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-) diff --git a/RELEASE_PROCEDURE.md b/RELEASE_PROCEDURE.md index 107682f7e..242900e51 100644 --- a/RELEASE_PROCEDURE.md +++ b/RELEASE_PROCEDURE.md @@ -19,9 +19,10 @@ 3. Perform thorough testing of the targeted revision to double down on stability [1] 4. Create an optimized build for comparing with the previous release. Changes in performance and/or measurement accuracy may need to be considered breaking. If in doubt, discuss! -5. If things look good, [File an issue](https://github.com/envoyproxy/nighthawk/issues/new?title=[VOTE]+Release+v0.x&body=Release%20X%20is%20ready%20for%20review.%20Please%20take%20a%20look%20and%20vote!) to raise a vote on publishing the drafted release, and point contributors to that. + TODO(#245): Formalize and/or automate this. +5. If things look good, [File an issue](https://github.com/envoyproxy/nighthawk/issues/new?title=%5BVOTE%5D+Release+v0.x.x&body=Release+v0.x.x%20is%20ready%20for%20review.%20Please%20take%20a%20look%20and%20vote!!) to raise a vote on publishing the drafted release, and point contributors to that. 6. Allow sufficient time to transpire for others to provide feedback, before moving on to the next step. If nobody raises blocking issues, it is time to go ahead and publish the drafted release. -7. |o/ Announce the new release on Slack, Twitter +7. |o/ Announce the new release on [Slack](https://envoyproxy.slack.com/archives/CDX3CGTT9), Twitter In case any of the steps above prevent shipping: @@ -35,5 +36,12 @@ is not tried and proven yet. (some of the integration tests are sensitive to tim unstable when slowed down due to concurrent runs). A sample command to do this: ```bash -bazel test --cache_test_results=no --test_env=ENVOY_IP_TEST_VERSIONS=all --runs_per_test=1000 --jobs 50 -c dbg --local_resources 20000,20,0.25 //test:* +bazel test \ + --cache_test_results=no \ + --test_env=ENVOY_IP_TEST_VERSIONS=all \ + --runs_per_test=1000 \ + --jobs 50 \ + -c dbg \ + --local_resources 20000,20,0.25 \ + //test:* ``` From e8ef55eb00e90b0037494fb285b5d55d1505b0e2 Mon Sep 17 00:00:00 2001 From: Otto van der Schaaf Date: Sat, 21 Dec 2019 09:25:21 +0100 Subject: [PATCH 4/6] Review feedback: drop twitter announcement channel Signed-off-by: Otto van der Schaaf --- RELEASE_PROCEDURE.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/RELEASE_PROCEDURE.md b/RELEASE_PROCEDURE.md index 242900e51..44e0f5586 100644 --- a/RELEASE_PROCEDURE.md +++ b/RELEASE_PROCEDURE.md @@ -22,7 +22,7 @@ TODO(#245): Formalize and/or automate this. 5. If things look good, [File an issue](https://github.com/envoyproxy/nighthawk/issues/new?title=%5BVOTE%5D+Release+v0.x.x&body=Release+v0.x.x%20is%20ready%20for%20review.%20Please%20take%20a%20look%20and%20vote!!) to raise a vote on publishing the drafted release, and point contributors to that. 6. Allow sufficient time to transpire for others to provide feedback, before moving on to the next step. If nobody raises blocking issues, it is time to go ahead and publish the drafted release. -7. |o/ Announce the new release on [Slack](https://envoyproxy.slack.com/archives/CDX3CGTT9), Twitter +7. |o/ Announce the new release on [Slack](https://envoyproxy.slack.com/archives/CDX3CGTT9). In case any of the steps above prevent shipping: From bb7db81f2ba9c354525029bbeea9a367fe7e43d3 Mon Sep 17 00:00:00 2001 From: Otto van der Schaaf Date: Thu, 9 Jan 2020 14:10:57 +0100 Subject: [PATCH 5/6] Fix version.h link Signed-off-by: Otto van der Schaaf --- RELEASE_PROCEDURE.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/RELEASE_PROCEDURE.md b/RELEASE_PROCEDURE.md index 44e0f5586..337d46e89 100644 --- a/RELEASE_PROCEDURE.md +++ b/RELEASE_PROCEDURE.md @@ -14,7 +14,7 @@ ## Release steps -1. Speculatively bump the version in [version.h](include/nighthawk/common/version.h) to the version you determined earlier. This may result in version gaps if a release attempt fails, but avoids having to freeze merges to master and/or having to work with release branches. In short it helps keeping the release procedure lean and mean and eliminates the need for blocking others while this procedure is in-flight. +1. Speculatively bump the version in [version_info.h](nighthawk/source/common/version_info.h) to the version you determined earlier. This may result in version gaps if a release attempt fails, but avoids having to freeze merges to master and/or having to work with release branches. In short it helps keeping the release procedure lean and mean and eliminates the need for blocking others while this procedure is in-flight. 2. Draft a [GitHub tagged release](https://github.com/envoyproxy/nighthawk/releases/new). Earlier releases are tagged like `v0.1`, but as of `0.3.0`we are using [semantic versioning](https://semver.org/spec/v2.0.0.html) 3. Perform thorough testing of the targeted revision to double down on stability [1] 4. Create an optimized build for comparing with the previous release. Changes in performance From 998c15e8233dd84cfe28824265115cb66e523e5b Mon Sep 17 00:00:00 2001 From: Otto van der Schaaf Date: Wed, 15 Jan 2020 01:26:37 +0100 Subject: [PATCH 6/6] Fix link to version_info.h Signed-off-by: Otto van der Schaaf --- RELEASE_PROCEDURE.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/RELEASE_PROCEDURE.md b/RELEASE_PROCEDURE.md index 337d46e89..e8a3542d0 100644 --- a/RELEASE_PROCEDURE.md +++ b/RELEASE_PROCEDURE.md @@ -14,7 +14,7 @@ ## Release steps -1. Speculatively bump the version in [version_info.h](nighthawk/source/common/version_info.h) to the version you determined earlier. This may result in version gaps if a release attempt fails, but avoids having to freeze merges to master and/or having to work with release branches. In short it helps keeping the release procedure lean and mean and eliminates the need for blocking others while this procedure is in-flight. +1. Speculatively bump the version in [version_info.h](source/common/version_info.h) to the version you determined earlier. This may result in version gaps if a release attempt fails, but avoids having to freeze merges to master and/or having to work with release branches. In short it helps keeping the release procedure lean and mean and eliminates the need for blocking others while this procedure is in-flight. 2. Draft a [GitHub tagged release](https://github.com/envoyproxy/nighthawk/releases/new). Earlier releases are tagged like `v0.1`, but as of `0.3.0`we are using [semantic versioning](https://semver.org/spec/v2.0.0.html) 3. Perform thorough testing of the targeted revision to double down on stability [1] 4. Create an optimized build for comparing with the previous release. Changes in performance