Skip to content

physical/spanner: use separate client for updating locks#9423

Merged
calvn merged 3 commits into
hashicorp:masterfrom
sethvargo:sethvargo/google_locks
Jul 9, 2020
Merged

physical/spanner: use separate client for updating locks#9423
calvn merged 3 commits into
hashicorp:masterfrom
sethvargo:sethvargo/google_locks

Conversation

@sethvargo
Copy link
Copy Markdown
Contributor

We believe this mitigates an issue where a large influx of requests cause the leader to be unable to update the lock table (since it cannot grab a client from the pool or the client has no more open connections), which causes cascading failure.

@sethvargo
Copy link
Copy Markdown
Contributor Author

Test output:

$ GOOGLE_SPANNER_DATABASE=projects/vs-test-282716/instances/test/databases/test GOOGLE_SPANNER_TABLE=Vault GOOGLE_SPANNER_HA_TABLE=VaultHA go test -v -count=1 ./physical/spanner/...
=== RUN   TestHABackend
2020-07-08T13:02:52.553-0400 [DEBUG] configuring backend
2020-07-08T13:02:52.553-0400 [DEBUG] creating HA client
2020-07-08T13:02:52.554-0400 [DEBUG] configuration: database=projects/vs-test-282716/instances/test/databases/test table=Vault haEnabled=true haTable=VaultHA maxParallel=0
2020-07-08T13:02:52.554-0400 [DEBUG] creating client
2020-07-08T13:02:52.554-0400 [DEBUG] configuring backend
2020-07-08T13:02:52.554-0400 [DEBUG] creating HA client
2020-07-08T13:02:52.555-0400 [DEBUG] configuration: database=projects/vs-test-282716/instances/test/databases/test table=Vault haEnabled=true haTable=VaultHA maxParallel=0
2020-07-08T13:02:52.555-0400 [DEBUG] creating client
--- PASS: TestHABackend (11.07s)
=== RUN   TestBackend
2020-07-08T13:03:03.113-0400 [DEBUG] configuring backend
2020-07-08T13:03:03.113-0400 [DEBUG] configuration: database=projects/vs-test-282716/instances/test/databases/test table=Vault haEnabled=false haTable=VaultHA maxParallel=0
2020-07-08T13:03:03.113-0400 [DEBUG] creating client
--- PASS: TestBackend (1.54s)
PASS
ok  	github.com/hashicorp/vault/physical/spanner	12.819s

@kalafut kalafut requested a review from calvn July 9, 2020 00:28
Copy link
Copy Markdown
Contributor

@calvn calvn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Small comments, otherwise 👍!

Comment thread physical/spanner/spanner.go Outdated
Comment thread physical/spanner/spanner.go Outdated
@sethvargo
Copy link
Copy Markdown
Contributor Author

@calvn updated. PTAL

sethvargo and others added 3 commits July 9, 2020 14:47
We believe this mitigates an issue where a large influx of requests
cause the leader to be unable to update the lock table (since it cannot
grab a client from the pool or the client has no more open connections),
which causes cascading failure.
@calvn calvn merged commit 29d1f21 into hashicorp:master Jul 9, 2020
calvn pushed a commit that referenced this pull request Jul 9, 2020
* physical/spanner: use separate client for updating locks

We believe this mitigates an issue where a large influx of requests
cause the leader to be unable to update the lock table (since it cannot
grab a client from the pool or the client has no more open connections),
which causes cascading failure.
calvn added a commit that referenced this pull request Jul 9, 2020
* physical/spanner: use separate client for updating locks

We believe this mitigates an issue where a large influx of requests
cause the leader to be unable to update the lock table (since it cannot
grab a client from the pool or the client has no more open connections),
which causes cascading failure.

Co-authored-by: Seth Vargo <seth@sethvargo.com>
andaley pushed a commit that referenced this pull request Jul 17, 2020
* physical/spanner: use separate client for updating locks

We believe this mitigates an issue where a large influx of requests
cause the leader to be unable to update the lock table (since it cannot
grab a client from the pool or the client has no more open connections),
which causes cascading failure.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants