Skip to content

gomod: fix btcd mempool leak fix and GetBlock heap escape #8401

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
guggero merged 4 commits into from
Jan 23, 2024
Merged

gomod: fix btcd mempool leak fix and GetBlock heap escape #8401

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
77e5dea
gomod: update `btcd` version to fix mem leak
yyforyongyu Jan 19, 2024
4ee862e
blockcache: fix heap escape in `GetBlock`
yyforyongyu Jan 21, 2024
ff1f33e
docs: update release notes for mem leak fix
yyforyongyu Jan 22, 2024
e6f5d17
chainreg: satisfy `chain.Interface` in `NoChainSource`
yyforyongyu Jan 23, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 7 additions & 3 deletions blockcache/blockcache.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,22 +52,26 @@ func (bc *BlockCache) GetBlock(hash *chainhash.Hash,
}

// Fetch the block from the chain backends.
Copy link
Copy Markdown
Collaborator

@ellemouton ellemouton Jan 22, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a couple of questions (just for my understanding):

  • are you saying that the pointer to the getBlockImpl gets stored on the heap? if so - im not sure i understand why that would be since we use it once here to produce a result & ie, we are not returning a pointer to the function right?
  • for making a copy of the block - I guess to me it only makes sense to make a copy of something before returning if the original object has references to some other info that is not needed once we return & so we want to get rid of those references. But here - are we removing any references? from the commit body it seems like you are saying that the block would have a reference to getBlockImpl but im not sure if that is true?
  • also - we would want the actual block to be on the heap right?
  • finally: did you notice this escape by eye or is there a tool you used to be able to identify this? asking cause my eye is definitely not familiar with this sort of gotcha

Copy link
Copy Markdown
Member Author

@yyforyongyu yyforyongyu Jan 22, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To answer,

  • nope not the getBlockImpl but the data it uses, hence the block.
  • def, we can copy it at difference places - either in btcd/rpcclient/chain.go's GetBlock, or here. As that being said, I see a lot more issues with heap escaping in btcd lib, kinda don't wanna touch it atm.
  • with the current design the block has to be on the heap because we store a pointer reference to that block in our cache. I have another local PR which removes the pointer references - don't think it helps tho, still digging as as you've said above, the block has to be cached.
  • this is a pure result from looking at the profile above, after fixing SendCmd, I was curious about why there was a heap allocation on GetBlock, then found this blockcache. Then got even more curious, as we only allow 20MB of cache size, but it used more. Then I took a closer look at the stacks,
(pprof) list GetBlock
Total: 115.90MB
ROUTINE ======================== github.com/btcsuite/btcd/rpcclient.(*Client).GetBlock in /Users/yong/Projects/lightning_labs/btcsuite/btcd/rpcclient/chain.go
         0    39.14MB (flat, cum) 33.77% of Total
         .          .    159:func (c *Client) GetBlock(blockHash *chainhash.Hash) (*wire.MsgBlock, error) {
         .    39.14MB    160:	return c.GetBlockAsync(blockHash).Receive()
         .          .    161:}
         .          .    162:
         .          .    163:// FutureGetBlockVerboseResult is a future promise to deliver the result of a
         .          .    164:// GetBlockVerboseAsync RPC invocation (or an applicable error).
         .          .    165:type FutureGetBlockVerboseResult struct {
ROUTINE ======================== github.com/btcsuite/btcd/rpcclient.FutureGetBlockResult.Receive in /Users/yong/Projects/lightning_labs/btcsuite/btcd/rpcclient/chain.go
         0    39.14MB (flat, cum) 33.77% of Total
         .          .    108:func (r FutureGetBlockResult) Receive() (*wire.MsgBlock, error) {
         .          .    109:	res, err := r.client.waitForGetBlockRes(r.Response, r.hash, false, false)
         .          .    110:	if err != nil {
         .          .    111:		return nil, err
         .          .    112:	}
         .          .    113:
         .          .    114:	// Unmarshal result as a string.
         .          .    115:	var blockHex string
         .          .    116:	err = json.Unmarshal(res, &blockHex)
         .          .    117:	if err != nil {
         .          .    118:		return nil, err
         .          .    119:	}
         .          .    120:
         .          .    121:	// Decode the serialized block hex to raw bytes.
         .          .    122:	serializedBlock, err := hex.DecodeString(blockHex)
         .          .    123:	if err != nil {
         .          .    124:		return nil, err
         .          .    125:	}
         .          .    126:
         .          .    127:	// Deserialize the block and return it.
         .          .    128:	var msgBlock wire.MsgBlock
         .    39.14MB    129:	err = msgBlock.Deserialize(bytes.NewReader(serializedBlock))
         .          .    130:	if err != nil {
         .          .    131:		return nil, err
         .          .    132:	}
         .          .    133:	return &msgBlock, nil
         .          .    134:}
ROUTINE ======================== github.com/btcsuite/btcwallet/chain.(*BitcoindClient).GetBlock in /Users/yong/Projects/lightning_labs/btcsuite/btcwallet/chain/bitcoind_client.go
         0    39.14MB (flat, cum) 33.77% of Total
         .          .    143:func (c *BitcoindClient) GetBlock(hash *chainhash.Hash) (*wire.MsgBlock, error) {
         .    39.14MB    144:	return c.chainConn.GetBlock(hash)
         .          .    145:}
         .          .    146:
         .          .    147:// GetBlockVerbose returns a verbose block from the hash.
         .          .    148:func (c *BitcoindClient) GetBlockVerbose(
         .          .    149:	hash *chainhash.Hash) (*btcjson.GetBlockVerboseResult, error) {
ROUTINE ======================== github.com/btcsuite/btcwallet/chain.(*BitcoindConn).GetBlock in /Users/yong/Projects/lightning_labs/btcsuite/btcwallet/chain/bitcoind_conn.go
         0    39.14MB (flat, cum) 33.77% of Total
         .          .    458:func (c *BitcoindConn) GetBlock(hash *chainhash.Hash) (*wire.MsgBlock, error) {
         .    39.14MB    459:	block, err := c.client.GetBlock(hash)
         .          .    460:	// Got the block from the backend successfully, return it.
         .          .    461:	if err == nil {
         .          .    462:		return block, nil
         .          .    463:	}
         .          .    464:
ROUTINE ======================== github.com/lightningnetwork/lnd/blockcache.(*BlockCache).GetBlock in /Users/yong/Projects/lightning_labs/lnd/blockcache/blockcache.go
         0    39.14MB (flat, cum) 33.77% of Total
         .          .     34:func (bc *BlockCache) GetBlock(hash *chainhash.Hash,
         .          .     35:	getBlockImpl func(hash *chainhash.Hash) (*wire.MsgBlock,
         .          .     36:		error)) (*wire.MsgBlock, error) {
         .          .     37:
         .          .     38:	bc.HashMutex.Lock(lntypes.Hash(*hash))
         .          .     39:	defer bc.HashMutex.Unlock(lntypes.Hash(*hash))
         .          .     40:
         .          .     41:	// Create an inv vector for getting the block.
         .          .     42:	inv := wire.NewInvVect(wire.InvTypeWitnessBlock, hash)
         .          .     43:
         .          .     44:	// Check if the block corresponding to the given hash is already
         .          .     45:	// stored in the blockCache and return it if it is.
         .          .     46:	cacheBlock, err := bc.Cache.Get(*inv)
         .          .     47:	if err != nil && err != cache.ErrElementNotFound {
         .          .     48:		return nil, err
         .          .     49:	}
         .          .     50:	if cacheBlock != nil {
         .          .     51:		return cacheBlock.MsgBlock(), nil
         .          .     52:	}
         .          .     53:
         .          .     54:	// Fetch the block from the chain backends.
         .    39.14MB     55:	block, err := getBlockImpl(hash)
         .          .     56:	if err != nil {
         .          .     57:		return nil, err
         .          .     58:	}
         .          .     59:
         .          .     60:	// Add the new block to blockCache. If the Cache is at its maximum
ROUTINE ======================== github.com/lightningnetwork/lnd/chainntnfs/bitcoindnotify.(*BitcoindNotifier).GetBlock in /Users/yong/Projects/lightning_labs/lnd/chainntnfs/bitcoindnotify/bitcoind.go
         0    23.10MB (flat, cum) 19.93% of Total
         .          .   1042:func (b *BitcoindNotifier) GetBlock(hash *chainhash.Hash) (*wire.MsgBlock,
         .          .   1043:	error) {
         .          .   1044:
         .    23.10MB   1045:	return b.blockCache.GetBlock(hash, b.chainConn.GetBlock)
         .          .   1046:}
         .          .   1047:
         .          .   1048:// SubscribeMempoolSpent allows the caller to register a subscription to watch
         .          .   1049:// for a spend of an outpoint in the mempool.The event will be dispatched once
         .          .   1050:// the outpoint is spent in the mempool.
ROUTINE ======================== github.com/lightningnetwork/lnd/lnwallet/btcwallet.(*BtcWallet).GetBlock in /Users/yong/Projects/lightning_labs/lnd/lnwallet/btcwallet/blockchain.go
         0    16.04MB (flat, cum) 13.84% of Total
         .          .    136:func (b *BtcWallet) GetBlock(blockHash *chainhash.Hash) (*wire.MsgBlock, error) {
         .          .    137:	_, ok := b.chain.(*chain.NeutrinoClient)
         .          .    138:	if !ok {
         .    16.04MB    139:		return b.blockCache.GetBlock(blockHash, b.chain.GetBlock)
         .          .    140:	}
         .          .    141:
         .          .    142:	// For the neutrino implementation of lnwallet.BlockChainIO the neutrino
         .          .    143:	// GetBlock function can be called directly since it uses the same block
         .          .    144:	// cache. However, it does not lock the block cache mutex for the given

And also with the command go build -gcflags='-m=3' . |& grep escapes to figure out which line escapes. That being said, I don't think we need to concern about this when writing code, optimization should always come later, as it can be an endless, but it's very difficult for me to resist the fun🤓

That being said, more analysis shows the Copy doesn't actually save us more memory in this case, it does give a better pattern to recognize the memory allocation, as now it's,

(pprof) list GetBlock
Total: 110.41MB
ROUTINE ======================== github.com/lightningnetwork/lnd/blockcache.(*BlockCache).GetBlock in /Users/yong/Projects/lightning_labs/lnd/blockcache/blockcache.go
         0    33.68MB (flat, cum) 30.51% of Total
         .          .     34:func (bc *BlockCache) GetBlock(hash *chainhash.Hash,
         .          .     35:	getBlockImpl func(hash *chainhash.Hash) (*wire.MsgBlock,
         .          .     36:		error)) (*wire.MsgBlock, error) {
         .          .     37:
         .          .     38:	bc.HashMutex.Lock(lntypes.Hash(*hash))
         .          .     39:	defer bc.HashMutex.Unlock(lntypes.Hash(*hash))
         .          .     40:
         .          .     41:	// Create an inv vector for getting the block.
         .          .     42:	inv := wire.NewInvVect(wire.InvTypeWitnessBlock, hash)
         .          .     43:
         .          .     44:	// Check if the block corresponding to the given hash is already
         .          .     45:	// stored in the blockCache and return it if it is.
         .          .     46:	cacheBlock, err := bc.Cache.Get(*inv)
         .          .     47:	if err != nil && err != cache.ErrElementNotFound {
         .          .     48:		return nil, err
         .          .     49:	}
         .          .     50:	if cacheBlock != nil {
         .          .     51:		return cacheBlock.MsgBlock(), nil
         .          .     52:	}
         .          .     53:
         .          .     54:	// Fetch the block from the chain backends.
         .          .     55:	msgBlock, err := getBlockImpl(hash)
         .          .     56:	if err != nil {
         .          .     57:		return nil, err
         .          .     58:	}
         .          .     59:
         .          .     60:	// Make a copy of the block so it won't escape to the heap.
         .    33.68MB     61:	msgBlockCopy := msgBlock.Copy()
         .          .     62:	block := btcutil.NewBlock(msgBlockCopy)
         .          .     63:
         .          .     64:	// Add the new block to blockCache. If the Cache is at its maximum
         .          .     65:	// capacity then the LFU item will be evicted in favour of this new
         .          .     66:	// block.
ROUTINE ======================== github.com/lightningnetwork/lnd/chainntnfs/bitcoindnotify.(*BitcoindNotifier).GetBlock in /Users/yong/Projects/lightning_labs/lnd/chainntnfs/bitcoindnotify/bitcoind.go
         0    33.68MB (flat, cum) 30.51% of Total
         .          .   1042:func (b *BitcoindNotifier) GetBlock(hash *chainhash.Hash) (*wire.MsgBlock,
         .          .   1043:	error) {
         .          .   1044:
         .    33.68MB   1045:	return b.blockCache.GetBlock(hash, b.chainConn.GetBlock)
         .          .   1046:}
         .          .   1047:
         .          .   1048:// SubscribeMempoolSpent allows the caller to register a subscription to watch
         .          .   1049:// for a spend of an outpoint in the mempool.The event will be dispatched once
         .          .   1050:// the outpoint is spent in the mempool.

block, err := getBlockImpl(hash)
msgBlock, err := getBlockImpl(hash)
if err != nil {
return nil, err
}

// Make a copy of the block so it won't escape to the heap.
msgBlockCopy := msgBlock.Copy()
block := btcutil.NewBlock(msgBlockCopy)

// Add the new block to blockCache. If the Cache is at its maximum
// capacity then the LFU item will be evicted in favour of this new
// block.
_, err = bc.Cache.Put(
*inv, &neutrino.CacheableBlock{
Block: btcutil.NewBlock(block),
Block: block,
},
)
if err != nil {
return nil, err
}

return block, nil
return msgBlockCopy, nil
}
7 changes: 7 additions & 0 deletions chainreg/no_chain_backend.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ import (
"errors"
"time"

"github.com/btcsuite/btcd/btcjson"
"github.com/btcsuite/btcd/btcutil"
"github.com/btcsuite/btcd/chaincfg/chainhash"
"github.com/btcsuite/btcd/wire"
Expand Down Expand Up @@ -213,4 +214,10 @@ func (n *NoChainSource) BackEnd() string {
return noChainBackendName
}

func (n *NoChainSource) TestMempoolAccept([]*wire.MsgTx,
float64) ([]*btcjson.TestMempoolAcceptResult, error) {

return nil, nil
}

var _ chain.Interface = (*NoChainSource)(nil)
5 changes: 5 additions & 0 deletions docs/release-notes/release-notes-0.17.4.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,10 @@
from [deadlocking](https://github.com/lightningnetwork/lnd/issues/8379)
the peer connection.

* [Fix](https://github.com/lightningnetwork/lnd/pull/8401) an issue that
caused memory leak for users running `lnd` with `bitcoind.rpcpolling=1`
mode.


# New Features
## Functional Enhancements
Expand All @@ -59,4 +63,5 @@
* Elle Mouton
* Keagan McClelland
* Olaoluwa Osuntokun
* Yong Yu
* ziggie1984
10 changes: 5 additions & 5 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,11 +4,11 @@ require (
github.com/NebulousLabs/go-upnp v0.0.0-20180202185039-29b680b06c82
github.com/Yawning/aez v0.0.0-20211027044916-e49e68abd344
github.com/andybalholm/brotli v1.0.3
github.com/btcsuite/btcd v0.23.5-0.20230905170901-80f5a0ffdf36
github.com/btcsuite/btcd v0.24.1-0.20240123000108-62e6af035ec5
github.com/btcsuite/btcd/btcec/v2 v2.3.2
github.com/btcsuite/btcd/btcutil v1.1.4-0.20230904040416-d4f519f5dc05
github.com/btcsuite/btcd/btcutil v1.1.5
github.com/btcsuite/btcd/btcutil/psbt v1.1.8
github.com/btcsuite/btcd/chaincfg/chainhash v1.0.2
github.com/btcsuite/btcd/chaincfg/chainhash v1.1.0
github.com/btcsuite/btclog v0.0.0-20170628155309-84c8d2346e9f
github.com/btcsuite/btcwallet v0.16.10-0.20231129183218-5df09dd43358
github.com/btcsuite/btcwallet/wallet/txauthor v1.3.2
Expand All @@ -20,7 +20,7 @@ require (
github.com/decred/dcrd/dcrec/secp256k1/v4 v4.0.1
github.com/go-errors/errors v1.0.1
github.com/golang-migrate/migrate/v4 v4.16.1
github.com/gorilla/websocket v1.4.2
github.com/gorilla/websocket v1.5.0
github.com/grpc-ecosystem/go-grpc-middleware v1.3.0
github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0
github.com/grpc-ecosystem/grpc-gateway/v2 v2.5.0
Expand Down Expand Up @@ -50,7 +50,7 @@ require (
github.com/miekg/dns v1.1.43
github.com/ory/dockertest/v3 v3.10.0
github.com/prometheus/client_golang v1.11.1
github.com/stretchr/testify v1.8.2
github.com/stretchr/testify v1.8.4
github.com/tv42/zbase32 v0.0.0-20160707012821-501572607d02
github.com/urfave/cli v1.22.9
go.etcd.io/etcd/client/pkg/v3 v3.5.7
Expand Down
22 changes: 11 additions & 11 deletions go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,10 +72,10 @@ github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6r
github.com/btcsuite/btcd v0.20.1-beta/go.mod h1:wVuoA8VJLEcwgqHBwHmzLRazpKxTv13Px/pDuV7OomQ=
github.com/btcsuite/btcd v0.22.0-beta.0.20220111032746-97732e52810c/go.mod h1:tjmYdS6MLJ5/s0Fj4DbLgSbDHbEqLJrtnHecBFkdz5M=
github.com/btcsuite/btcd v0.22.0-beta.0.20220204213055-eaf0459ff879/go.mod h1:osu7EoKiL36UThEgzYPqdRaxeo0NU8VoXqgcnwpey0g=
github.com/btcsuite/btcd v0.23.0/go.mod h1:0QJIIN1wwIXF/3G/m87gIwGniDMDQqjVn4SZgnFpsYY=
github.com/btcsuite/btcd v0.23.1/go.mod h1:0QJIIN1wwIXF/3G/m87gIwGniDMDQqjVn4SZgnFpsYY=
github.com/btcsuite/btcd v0.23.5-0.20230905170901-80f5a0ffdf36 h1:g/UbZ6iSzcUH9kEvC+rB8UBCqahmt69e8y6nCegczbg=
github.com/btcsuite/btcd v0.23.5-0.20230905170901-80f5a0ffdf36/go.mod h1:0QJIIN1wwIXF/3G/m87gIwGniDMDQqjVn4SZgnFpsYY=
github.com/btcsuite/btcd v0.23.5-0.20231215221805-96c9fd8078fd/go.mod h1:nm3Bko6zh6bWP60UxwoT5LzdGJsQJaPo6HjduXq9p6A=
github.com/btcsuite/btcd v0.24.1-0.20240123000108-62e6af035ec5 h1:8BHBWvtP6kkzvmCpyWEznq4eS0gfLOSVuXLesv413Xs=
github.com/btcsuite/btcd v0.24.1-0.20240123000108-62e6af035ec5/go.mod h1:5C8ChTkl5ejr3WHj8tkQSCmydiMEPB0ZhQhehpq7Dgg=
github.com/btcsuite/btcd/btcec/v2 v2.1.0/go.mod h1:2VzYrv4Gm4apmbVVsSq5bqf1Ec8v56E48Vt0Y/umPgA=
github.com/btcsuite/btcd/btcec/v2 v2.1.1/go.mod h1:ctjw4H1kknNJmRN4iP1R7bTQ+v3GJkZBd6mui8ZsAZE=
github.com/btcsuite/btcd/btcec/v2 v2.1.3/go.mod h1:ctjw4H1kknNJmRN4iP1R7bTQ+v3GJkZBd6mui8ZsAZE=
Expand All @@ -84,14 +84,14 @@ github.com/btcsuite/btcd/btcec/v2 v2.3.2/go.mod h1:zYzJ8etWJQIv1Ogk7OzpWjowwOdXY
github.com/btcsuite/btcd/btcutil v1.0.0/go.mod h1:Uoxwv0pqYWhD//tfTiipkxNfdhG9UrLwaeswfjfdF0A=
github.com/btcsuite/btcd/btcutil v1.1.0/go.mod h1:5OapHB7A2hBBWLm48mmw4MOHNJCcUBTwmWH/0Jn8VHE=
github.com/btcsuite/btcd/btcutil v1.1.1/go.mod h1:nbKlBMNm9FGsdvKvu0essceubPiAcI57pYBNnsLAa34=
github.com/btcsuite/btcd/btcutil v1.1.4-0.20230904040416-d4f519f5dc05 h1:aemxF+69pT9sYC5E6Qj71zQVHcF72m0BNcVhCl3/thU=
github.com/btcsuite/btcd/btcutil v1.1.4-0.20230904040416-d4f519f5dc05/go.mod h1:UR7dsSJzJUfMmFiiLlIrMq1lS9jh9EdCV7FStZSnpi0=
github.com/btcsuite/btcd/btcutil v1.1.5 h1:+wER79R5670vs/ZusMTF1yTcRYE5GUsFbdjdisflzM8=
github.com/btcsuite/btcd/btcutil v1.1.5/go.mod h1:PSZZ4UitpLBWzxGd5VGOrLnmOjtPP/a6HaFo12zMs00=
github.com/btcsuite/btcd/btcutil/psbt v1.1.8 h1:4voqtT8UppT7nmKQkXV+T9K8UyQjKOn2z/ycpmJK8wg=
github.com/btcsuite/btcd/btcutil/psbt v1.1.8/go.mod h1:kA6FLH/JfUx++j9pYU0pyu+Z8XGBQuuTmuKYUf6q7/U=
github.com/btcsuite/btcd/chaincfg/chainhash v1.0.0/go.mod h1:7SFka0XMvUgj3hfZtydOrQY2mwhPclbT2snogU7SQQc=
github.com/btcsuite/btcd/chaincfg/chainhash v1.0.1/go.mod h1:7SFka0XMvUgj3hfZtydOrQY2mwhPclbT2snogU7SQQc=
github.com/btcsuite/btcd/chaincfg/chainhash v1.0.2 h1:KdUfX2zKommPRa+PD0sWZUyXe9w277ABlgELO7H04IM=
github.com/btcsuite/btcd/chaincfg/chainhash v1.0.2/go.mod h1:7SFka0XMvUgj3hfZtydOrQY2mwhPclbT2snogU7SQQc=
github.com/btcsuite/btcd/chaincfg/chainhash v1.1.0 h1:59Kx4K6lzOW5w6nFlA0v5+lk/6sjybR934QNHSJZPTQ=
github.com/btcsuite/btcd/chaincfg/chainhash v1.1.0/go.mod h1:7SFka0XMvUgj3hfZtydOrQY2mwhPclbT2snogU7SQQc=
github.com/btcsuite/btclog v0.0.0-20170628155309-84c8d2346e9f h1:bAs4lUbRJpnnkd9VhRV3jjAVU7DJVjMaK+IsvSeZvFo=
github.com/btcsuite/btclog v0.0.0-20170628155309-84c8d2346e9f/go.mod h1:TdznJufoqS23FtqVCzL0ZqgP5MqXbb4fg/WgDys70nA=
github.com/btcsuite/btcutil v0.0.0-20190425235716-9e5f4b9a998d/go.mod h1:+5NJ2+qvTyV9exUAL/rxXi3DcLg2Ts+ymUAY5y4NvMg=
Expand Down Expand Up @@ -295,8 +295,8 @@ github.com/google/uuid v1.3.0 h1:t6JiXgmwXMjEs8VusXIJk2BXHsn+wx8BZdTaoZ5fu7I=
github.com/google/uuid v1.3.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/googleapis/gax-go/v2 v2.0.4/go.mod h1:0Wqv26UfaUD9n4G6kQubkQ+KchISgw+vpHVxEJEs9eg=
github.com/googleapis/gax-go/v2 v2.0.5/go.mod h1:DWXyrwAJ9X0FpwwEdw+IPEYBICEFu5mhpdKc/us6bOk=
github.com/gorilla/websocket v1.4.2 h1:+/TMaTYc4QFitKJxsQ7Yye35DkWvkdLcvGKqM+x0Ufc=
github.com/gorilla/websocket v1.4.2/go.mod h1:YR8l580nyteQvAITg2hZ9XVh4b55+EU/adAjf1fMHhE=
github.com/gorilla/websocket v1.5.0 h1:PPwGk2jz7EePpoHN/+ClbZu8SPxiqlu12wZP/3sWmnc=
github.com/gorilla/websocket v1.5.0/go.mod h1:YR8l580nyteQvAITg2hZ9XVh4b55+EU/adAjf1fMHhE=
github.com/grpc-ecosystem/go-grpc-middleware v1.3.0 h1:+9834+KizmvFV7pXQGSXQTsaWhq2GjuNUt0aUU0YBYw=
github.com/grpc-ecosystem/go-grpc-middleware v1.3.0/go.mod h1:z0ButlSOZa5vEBq9m2m2hlwIgKw+rp3sdCBRoJY+30Y=
github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0 h1:Ovs26xHkKqVztRpIrF/92BcuyuQ/YW4NSIpoGtfXNho=
Expand Down Expand Up @@ -604,8 +604,8 @@ github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/
github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU=
github.com/stretchr/testify v1.8.1/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4=
github.com/stretchr/testify v1.8.2 h1:+h33VjcLVPDHtOdpUCuF+7gSuG3yGIftsP1YvFihtJ8=
github.com/stretchr/testify v1.8.2/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4=
github.com/stretchr/testify v1.8.4 h1:CcVxjf3Q8PM0mHUKJCdn+eZZtm5yQwehR5yeSVQQcUk=
github.com/stretchr/testify v1.8.4/go.mod h1:sz/lmYIOXD/1dqDmKjjqLyZ2RngseejIcXlSw2iwfAo=
github.com/syndtr/gocapability v0.0.0-20200815063812-42c35b437635/go.mod h1:hkRG7XYTFWNJGYcbNJQlaLq0fg1yr4J4t/NcTQtrfww=
github.com/syndtr/goleveldb v1.0.1-0.20210819022825-2ae1ddf74ef7 h1:epCh84lMvA70Z7CTTCmYQn2CKbY8j86K7/FAIr141uY=
github.com/syndtr/goleveldb v1.0.1-0.20210819022825-2ae1ddf74ef7/go.mod h1:q4W45IWZaF22tdD+VEXcAWRA037jwmWEB5VWYORlTpc=
Expand Down