Skip to content

[NET-6741] make: Add target for updating dependencies across all modules#19785

Merged
zalimeni merged 1 commit into
mainfrom
zalimeni/net-6741-add-make-target-dependency-update-modules
Feb 21, 2024
Merged

[NET-6741] make: Add target for updating dependencies across all modules#19785
zalimeni merged 1 commit into
mainfrom
zalimeni/net-6741-add-make-target-dependency-update-modules

Conversation

@zalimeni
Copy link
Copy Markdown
Member

@zalimeni zalimeni commented Nov 30, 2023 •
edited
Loading

To enable more consistent and error-proof dependency management, add a Make target that will set a dependency version across all submodules that require it.

Also runs go mod tidy. This first ensures the dependency addition is reverted if the module in question does not require it; it also ensures that any additional cleanup needed in go.mod/go.sum is applied.

Note on approach vs. go.work

At one point, it seemed we could avoid executing go get per each go.mod by instead moving to a versioned go.work and using workspace-based dependency management commands. Though there's still reasons to move to go.work beyond this change, it probably won't work for bumping dependencies conveniently: go work sync consolidates versions using MVS, so in practice, it seems we still need to use a visit-all approach to be able to blindly bump a given dependency only where it’s used. go work sync simply enforces consistency across all go.mod files once any go.mod is updated (docs):

The syncing is done by sequentially upgrading each of the dependency modules specified in a workspace module to the version in the build list if the dependency module's version is not already the same as the build list's version. Note that Minimal Version Selection guarantees that the build list's version of each module is always the same or higher than that in each workspace module.

Description

Simplify dependency management, particularly for addressing CVEs where a consistent minimum version is required.

Example run:

❯ make go-mod-get DEP_VERSION=github.com/hashicorp/raft@v1.6.0
--> Running go get github.com/hashicorp/raft@v1.6.0 (.)
go: upgraded github.com/hashicorp/go-msgpack/v2 v2.0.0 => v2.1.1
go: upgraded github.com/hashicorp/raft v1.5.0 => v1.6.0
go: upgraded github.com/stretchr/testify v1.8.3 => v1.8.4
go: upgraded golang.org/x/mod v0.12.0 => v0.13.0
go: upgraded golang.org/x/sync v0.3.0 => v0.4.0
go: upgraded golang.org/x/tools v0.12.1-0.20230815132531-74c255bcf846 => v0.14.0
--> Running go mod tidy (.)
--> Running go get github.com/hashicorp/raft@v1.6.0 (./api)
go: added github.com/hashicorp/go-msgpack/v2 v2.1.1
go: added github.com/hashicorp/raft v1.6.0
go: upgraded github.com/stretchr/testify v1.8.3 => v1.8.4
--> Running go mod tidy (./api)
--> Running go get github.com/hashicorp/raft@v1.6.0 (./envoyextensions)
go: added github.com/hashicorp/go-msgpack/v2 v2.1.1
go: added github.com/hashicorp/raft v1.6.0
go: upgraded github.com/stretchr/testify v1.8.3 => v1.8.4
--> Running go mod tidy (./envoyextensions)
--> Running go get github.com/hashicorp/raft@v1.6.0 (./internal/tools/proto-gen-rpc-glue)
go: added github.com/armon/go-metrics v0.4.1
go: added github.com/fatih/color v1.13.0
go: added github.com/hashicorp/go-hclog v1.5.0
go: added github.com/hashicorp/go-immutable-radix v1.0.0
go: added github.com/hashicorp/go-msgpack/v2 v2.1.1
go: added github.com/hashicorp/golang-lru v0.5.0
go: added github.com/hashicorp/raft v1.6.0
go: added github.com/mattn/go-colorable v0.1.12
go: added github.com/mattn/go-isatty v0.0.14
go: upgraded github.com/stretchr/testify v1.8.3 => v1.8.4
go: added golang.org/x/sys v0.13.0
--> Running go mod tidy (./internal/tools/proto-gen-rpc-glue)
--> Running go get github.com/hashicorp/raft@v1.6.0 (./internal/tools/protoc-gen-consul-rate-limit)
go: added github.com/armon/go-metrics v0.4.1
go: added github.com/fatih/color v1.13.0
go: added github.com/hashicorp/go-hclog v1.5.0
go: added github.com/hashicorp/go-immutable-radix v1.0.0
go: added github.com/hashicorp/go-msgpack/v2 v2.1.1
go: added github.com/hashicorp/golang-lru v0.5.0
go: added github.com/hashicorp/raft v1.6.0
go: added github.com/mattn/go-colorable v0.1.12
go: added github.com/mattn/go-isatty v0.0.14
--> Running go mod tidy (./internal/tools/protoc-gen-consul-rate-limit)
--> Running go get github.com/hashicorp/raft@v1.6.0 (./proto-public)
go: added github.com/armon/go-metrics v0.4.1
go: added github.com/fatih/color v1.13.0
go: added github.com/hashicorp/go-hclog v1.5.0
go: added github.com/hashicorp/go-immutable-radix v1.0.0
go: added github.com/hashicorp/go-msgpack/v2 v2.1.1
go: added github.com/hashicorp/golang-lru v0.5.0
go: added github.com/hashicorp/raft v1.6.0
go: added github.com/mattn/go-colorable v0.1.12
go: added github.com/mattn/go-isatty v0.0.14
go: upgraded github.com/stretchr/testify v1.8.3 => v1.8.4
--> Running go mod tidy (./proto-public)
--> Running go get github.com/hashicorp/raft@v1.6.0 (./sdk)
go: added github.com/armon/go-metrics v0.4.1
go: added github.com/hashicorp/go-immutable-radix v1.0.0
go: added github.com/hashicorp/go-msgpack/v2 v2.1.1
go: added github.com/hashicorp/golang-lru v0.5.0
go: added github.com/hashicorp/raft v1.6.0
go: upgraded github.com/stretchr/testify v1.8.3 => v1.8.4
--> Running go mod tidy (./sdk)
--> Running go get github.com/hashicorp/raft@v1.6.0 (./test-integ)
go: upgraded golang.org/x/mod v0.12.0 => v0.13.0
go: upgraded golang.org/x/tools v0.12.1-0.20230815132531-74c255bcf846 => v0.14.0
--> Running go mod tidy (./test-integ)
--> Running go get github.com/hashicorp/raft@v1.6.0 (./test/integration/connect/envoy/test-sds-server)
go: upgraded github.com/hashicorp/raft v1.4.0 => v1.6.0
--> Running go mod tidy (./test/integration/connect/envoy/test-sds-server)
--> Running go get github.com/hashicorp/raft@v1.6.0 (./test/integration/consul-container)
go: upgraded github.com/hashicorp/raft v1.5.0 => v1.6.0
go: upgraded golang.org/x/mod v0.12.0 => v0.13.0
go: upgraded golang.org/x/sync v0.3.0 => v0.4.0
go: upgraded golang.org/x/tools v0.12.1-0.20230815132531-74c255bcf846 => v0.14.0
--> Running go mod tidy (./test/integration/consul-container)
--> Running go get github.com/hashicorp/raft@v1.6.0 (./test/integration/consul-container/test/envoy_extensions/testdata/wasm_test_files)
go: added github.com/armon/go-metrics v0.4.1
go: added github.com/fatih/color v1.13.0
go: added github.com/hashicorp/go-hclog v1.5.0
go: added github.com/hashicorp/go-immutable-radix v1.0.0
go: added github.com/hashicorp/go-msgpack/v2 v2.1.1
go: added github.com/hashicorp/golang-lru v0.5.0
go: added github.com/hashicorp/raft v1.6.0
go: added github.com/mattn/go-colorable v0.1.12
go: added github.com/mattn/go-isatty v0.0.14
go: added golang.org/x/sys v0.13.0
--> Running go mod tidy (./test/integration/consul-container/test/envoy_extensions/testdata/wasm_test_files)
--> Running go get github.com/hashicorp/raft@v1.6.0 (./testing/deployer)
go: added github.com/hashicorp/go-msgpack/v2 v2.1.1
go: added github.com/hashicorp/raft v1.6.0
go: upgraded github.com/stretchr/testify v1.8.3 => v1.8.4
--> Running go mod tidy (./testing/deployer)
--> Running go get github.com/hashicorp/raft@v1.6.0 (./troubleshoot)
go: added github.com/hashicorp/go-msgpack/v2 v2.1.1
go: added github.com/hashicorp/raft v1.6.0
go: upgraded github.com/stretchr/testify v1.8.3 => v1.8.4
--> Running go mod tidy (./troubleshoot)

❯ git status
On branch zalimeni/add-make-target-dependency-update-modules
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
	modified:   api/go.mod
	modified:   api/go.sum
	modified:   envoyextensions/go.mod
	modified:   envoyextensions/go.sum
	modified:   go.mod
	modified:   go.sum
	modified:   internal/tools/proto-gen-rpc-glue/go.mod
	modified:   internal/tools/proto-gen-rpc-glue/go.sum
	modified:   proto-public/go.mod
	modified:   proto-public/go.sum
	modified:   sdk/go.mod
	modified:   sdk/go.sum
	modified:   test-integ/go.mod
	modified:   test-integ/go.sum
	modified:   test/integration/connect/envoy/test-sds-server/go.mod
	modified:   test/integration/connect/envoy/test-sds-server/go.sum
	modified:   test/integration/consul-container/go.mod
	modified:   test/integration/consul-container/go.sum
	modified:   test/integration/consul-container/test/envoy_extensions/testdata/wasm_test_files/go.mod
	modified:   test/integration/consul-container/test/envoy_extensions/testdata/wasm_test_files/go.sum
	modified:   testing/deployer/go.mod
	modified:   testing/deployer/go.sum
	modified:   troubleshoot/go.mod
	modified:   troubleshoot/go.sum

PR Checklist

  • updated test coverage
  • external facing docs updated
  • appropriate backport labels added
  • not a security concern

@zalimeni zalimeni added pr/no-changelog PR does not need a corresponding .changelog entry backport/1.15 labels Nov 30, 2023
@zalimeni
Copy link
Copy Markdown
Member Author

zalimeni commented Nov 30, 2023

After talking w/ Matt K., we think go.work would be a nice solution w/ added benefits for replace directive removal, so going to give that a shot intead.

@zalimeni zalimeni closed this Nov 30, 2023
@zalimeni
Copy link
Copy Markdown
Member Author

zalimeni commented Feb 21, 2024

Update: go work sync consolidates versions using MVS, so in practice, it seems we still need to use a visit-all approach to be able to blindly bump a given dependency only where it’s used. go work sync simply enforces consistency across all go.mod files once any go.mod is updated (docs):

The syncing is done by sequentially upgrading each of the dependency modules specified in a workspace module to the version in the build list if the dependency module's version is not already the same as the build list's version. Note that Minimal Version Selection guarantees that the build list's version of each module is always the same or higher than that in each workspace module.

@zalimeni zalimeni reopened this Feb 21, 2024
@zalimeni
make: Add target for updating dependencies across all modules
6f0e24c 
To enable more consistent and error-proof dependency management, add a
Make target that will set a dependency version across all submodules
that require it.

Also runs `go mod tidy`. This first ensures the dependency addition is
reverted if the module in question does not require it; it also ensures
that any additional cleanup needed in `go.mod`/`go.sum` is applied.
@zalimeni zalimeni force-pushed the zalimeni/net-6741-add-make-target-dependency-update-modules branch from c0704a6 to 6f0e24c Compare February 21, 2024 14:27
@zalimeni zalimeni marked this pull request as ready for review February 21, 2024 14:41
@zalimeni zalimeni merged commit 858691b into main Feb 21, 2024
@zalimeni zalimeni deleted the zalimeni/net-6741-add-make-target-dependency-update-modules branch February 21, 2024 16:11
This was referenced Feb 21, 2024
Closed
Merged
Merged
Merged
This was referenced Feb 21, 2024
Merged
Merged
This was referenced Feb 21, 2024
Merged
Merged
Merged
Merged
@zalimeni zalimeni mentioned this pull request Feb 21, 2024
Merged
This was referenced Feb 21, 2024
Merged
Merged
Merged
Merged
@hc-github-team-consul-core
Copy link
Copy Markdown
Collaborator

hc-github-team-consul-core commented Feb 22, 2024

@zalimeni, a backport is missing for this PR [19785] for versions [1.15] please perform the backport manually and add the following snippet to your backport PR description:

<details>
	<summary> Overview of commits </summary>
		- <<backport commit 1>>
		- <<backport commit 2>>
		...
</details>

5 similar comments
@hc-github-team-consul-core
Copy link
Copy Markdown
Collaborator

hc-github-team-consul-core commented Feb 23, 2024

@zalimeni, a backport is missing for this PR [19785] for versions [1.15] please perform the backport manually and add the following snippet to your backport PR description:

<details>
	<summary> Overview of commits </summary>
		- <<backport commit 1>>
		- <<backport commit 2>>
		...
</details>

@hc-github-team-consul-core
Copy link
Copy Markdown
Collaborator

hc-github-team-consul-core commented Feb 24, 2024

@zalimeni, a backport is missing for this PR [19785] for versions [1.15] please perform the backport manually and add the following snippet to your backport PR description:

<details>
	<summary> Overview of commits </summary>
		- <<backport commit 1>>
		- <<backport commit 2>>
		...
</details>

@hc-github-team-consul-core
Copy link
Copy Markdown
Collaborator

hc-github-team-consul-core commented Feb 25, 2024

@zalimeni, a backport is missing for this PR [19785] for versions [1.15] please perform the backport manually and add the following snippet to your backport PR description:

<details>
	<summary> Overview of commits </summary>
		- <<backport commit 1>>
		- <<backport commit 2>>
		...
</details>

@hc-github-team-consul-core
Copy link
Copy Markdown
Collaborator

hc-github-team-consul-core commented Feb 26, 2024

@zalimeni, a backport is missing for this PR [19785] for versions [1.15] please perform the backport manually and add the following snippet to your backport PR description:

<details>
	<summary> Overview of commits </summary>
		- <<backport commit 1>>
		- <<backport commit 2>>
		...
</details>

@hc-github-team-consul-core
Copy link
Copy Markdown
Collaborator

hc-github-team-consul-core commented Feb 27, 2024

@zalimeni, a backport is missing for this PR [19785] for versions [1.15] please perform the backport manually and add the following snippet to your backport PR description:

<details>
	<summary> Overview of commits </summary>
		- <<backport commit 1>>
		- <<backport commit 2>>
		...
</details>

@hc-github-team-consul-core hc-github-team-consul-core mentioned this pull request Feb 27, 2024
Closed
4 tasks
This was referenced Sep 27, 2024
Open
Open
Open
Open
Open
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@mkeeler mkeeler mkeeler approved these changes

@johnlanda johnlanda Awaiting requested review from johnlanda

Assignees

No one assigned

Labels

pr/no-changelog PR does not need a corresponding .changelog entry pr/no-metrics-test

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@zalimeni @hc-github-team-consul-core @mkeeler @jmurret