Initial commit for producer/consumer loadtest client by philipsu522 · Pull Request #1190 · sei-protocol/sei-chain

philipsu522 · 2023-12-28T00:01:14Z

Describe your changes and provide context

Loadtest improvements:

Move to a producer consumer model
Use random numbers for seqno (need to merge this pr to Add config to disable seqno sei-cosmos#385)
Allow for long running tests
Remove writing data to a "result" file that then needs to be queried with another tool, instead generate stats on the fly

Testing performed to validate your change

Ran on loadtest cluster

loadtest/main.go

philipsu522 · 2023-12-28T19:28:20Z

go.mod

 	github.com/CosmWasm/wasmd => github.com/sei-protocol/sei-wasmd v0.0.4
 	github.com/confio/ics23/go => github.com/cosmos/cosmos-sdk/ics23/go v0.8.0
-	github.com/cosmos/cosmos-sdk => github.com/sei-protocol/sei-cosmos v0.2.68
+	github.com/cosmos/cosmos-sdk => github.com/sei-protocol/sei-cosmos v0.2.69-0.20231228001217-1e240574c003


will revert go.mod changes once i merge sei-protocol/sei-cosmos#385

udpatil · 2023-12-29T20:28:06Z

loadtest/loadtest_client.go

+				})
+				// Use random seqno to get around txs that might already be seen in mempool
+
+				c.SignerClient.SignTx(c.ChainID, &txBuilder, key, uint64(rand.Intn(math.MaxInt)))


is this necessary after disabling seqno check? couldn't we just set them all to a constant seqno?

I think you probably have to at least make these transactions different because mempool will reject duplicate txs. However, even random seq can still have a chance to result in duplicates, we should probably just switch to a monitonically increasing seqno here?

I wonder whether we need to apply rate limit for signing tx or not at all. We should apply the rate limiting only for the broadcast tx right? For signing txs, we can keep signing and generating tx as long as the queue/channel is not full

udpatil · 2023-12-29T20:29:07Z

loadtest/loadtest_client.go

+			if !ok {
+				fmt.Printf("Stopping consumer %d\n", consumerId)
+			}
+			SendTx(tx, typestx.BroadcastMode_BROADCAST_MODE_SYNC, false, *c, sentCount)


we likely want this to be async, right? otherwise there would be a bottleneck for sending txs since theyll need to sequentially pass checkTx to enter mempool?

We want to do sync here and wait until the response back, otherwise you have to query the tx and figure out when it is completed. We can put each tx into a separate goroutine though

udpatil · 2023-12-29T20:38:01Z

loadtest/config.json

@@ -5,7 +5,7 @@
    "tls": false,
    "msgs_per_tx": 10,


Note, we may want to change this default to like 1, since otherwise it will send a TON of messages, whereas before the config fields were named weirdly such that txs per block = 500 and msgs per tx = 10 would result in 50 txs with 10 messages each IIRC. now, it would ACTUALLY result in 500TPS of 10 messages PER tx

Did we change the concept here? I think previously if target is 500, batch size is 10, then it will only send 50txs and each tx will contain 10 messages

logic has not changed. will set this to 1 as default

yzang2019 · 2024-01-02T19:20:33Z

loadtest/loadtest_client.go

+		case <-done:
+			fmt.Printf("Stopping producer %d\n", producerId)
+			return
+		case <-ticker.C:


Instead of using ticker to do rate limiting, I think a better approach is just to use "golang.org/x/time/rate": https://github.com/sei-protocol/dex-indexer/blob/main/cmd/index_producer/main.go#L63

yzang2019 · 2024-01-02T19:21:55Z

loadtest/loadtest_client.go

+			if !ok {
+				fmt.Printf("Stopping consumer %d\n", consumerId)
+			}
+			SendTx(tx, typestx.BroadcastMode_BROADCAST_MODE_SYNC, false, *c, sentCount)


We want to do sync here and wait until the response back, otherwise you have to query the tx and figure out when it is completed. We can put each tx into a separate goroutine though

yzang2019 · 2024-01-02T19:23:43Z

loadtest/config.json

@@ -5,7 +5,7 @@
    "tls": false,
    "msgs_per_tx": 10,


Did we change the concept here? I think previously if target is 500, batch size is 10, then it will only send 50txs and each tx will contain 10 messages

philipsu522 · 2024-01-09T17:15:58Z

loadtest/main.go

+	elapsed := time.Since(startTime)
+	produced := atomic.LoadInt64(producedCount)
+	sent := atomic.LoadInt64(sentCount)
+	tps := float64(sent-*prevSentCount) / elapsed.Seconds()


b/c sender uses broadcast mode = block, we can use client-side metrics to calculate tps

* Initial commit for producer/consumer loadtest client (#1190) * Initial commit for producer/consumer loadtest client * update sei-cosmos * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * rm rounds * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * finalize * debug * debug * debug * debug * debug * debug * debug * debug * bump cosmos * gofmt * debug * debug * Bump version to v3.6.1 (#1222) * Bump version to v3.6.0 * Fix empty branch * Bump cosmos to fix upgrade migration * Update version * Fix changelogs readme * Bump tendermint version * Fix version * Add 3.6.1 upgrade handler * Supress lint * Add migration process readme for SeiDB (#1221) * Add seidb migration steps * Add migration process for SeiDB * Fix * Update seidb_migration.md * Loadtest producer consumer various fixes (#1227) * debug * debug * debug * set default broadcast to sync * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * grpc * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * lint * int * Add TransferFrom method to contract and bindings * Lt client fix (#1260) * Fix account loading in loadtest client * Fix account loading in loadtest client * debug * debug * debug * debug * debug * debug * debug * debug * update * base * base with error * comments * fix bug * old additions * revert merge * pre merge * issues * add tests for transfer and transferFrom * add approve functions * add send_from --------- Co-authored-by: Philip Su <philip.su.522@gmail.com> Co-authored-by: Yiming Zang <50607998+yzang2019@users.noreply.github.com> Co-authored-by: Mingjun <mj@mingjuns-mbp.lan>

* Initial commit for producer/consumer loadtest client (#1190) * Initial commit for producer/consumer loadtest client * update sei-cosmos * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * rm rounds * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * finalize * debug * debug * debug * debug * debug * debug * debug * debug * bump cosmos * gofmt * debug * debug * Bump version to v3.6.1 (#1222) * Bump version to v3.6.0 * Fix empty branch * Bump cosmos to fix upgrade migration * Update version * Fix changelogs readme * Bump tendermint version * Fix version * Add 3.6.1 upgrade handler * Supress lint * Add migration process readme for SeiDB (#1221) * Add seidb migration steps * Add migration process for SeiDB * Fix * Update seidb_migration.md * Loadtest producer consumer various fixes (#1227) * debug * debug * debug * set default broadcast to sync * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * grpc * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * debug * lint * int * Lt client fix (#1260) * Fix account loading in loadtest client * Fix account loading in loadtest client * debug * debug * debug * debug * debug * debug * debug * debug * update * SeiDB State Store Integration Tests (#1242) * SeiDB State Store Integration Tests * Update to differences * Update num contracts/denoms + jq * change length * reduce sleeps * Check reverse * logging * Update block height logging * update seid command * Update * remove logging * Update to branch * Update bank tests * Point to latest seidb --------- Co-authored-by: kbhat1 <kartikbhat@Kartiks-MacBook-Pro-2.local> Co-authored-by: kbhat1 <kartikbhat@kartiks-mbp-2.mynetworksettings.com> * Update synchronous execution to set tx indices properly (#1283) * fix debug iavl dump (#1287) * Keep v prefix for seid version (#1311) keep v prefix for seid version * Release v3.6.3 (#1317) * [SeiDB] Add integration test for historical bank send (#1329) * Add integration test for historical bank send * Fix go mod * Ignore error for vote already exist (#1325) * Ignore error for vote already exist * Fix go mod * Add error log --------- Co-authored-by: Philip Su <philip.su.522@gmail.com> Co-authored-by: Yiming Zang <50607998+yzang2019@users.noreply.github.com> Co-authored-by: Kartik Bhat <kartikbhatri@gmail.com> Co-authored-by: kbhat1 <kartikbhat@Kartiks-MacBook-Pro-2.local> Co-authored-by: kbhat1 <kartikbhat@kartiks-mbp-2.mynetworksettings.com> Co-authored-by: Uday Patil <udpatil@gmail.com>