Add historical state offload stream hook by Kbhat1 · Pull Request #3183 · sei-protocol/sei-chain

Kbhat1 · 2026-04-03T14:11:07Z

Describe your changes and provide context

Add a cryptosim historical offload benchmark backend that publishes changelog entries directly to Kafka instead of writing local state
Support configurable Kafka batching/auth settings, including AWS MSK IAM, so the benchmark matches the intended cloud offload flow
Keep the benchmark write-only and Kafka-only to measure pure offload throughput with minimal extra benchmark-path code

Testing performed to validate your change

Unit tests
Verified cryptosim run with MSK

Introduce a transport-agnostic changelog stream interface so historical state can be published to an external queue and replayed later without coupling the state store to a specific message system. Made-with: Cursor

Add a write-only cryptosim backend that exercises the SS historical offload hook with a buffered in-process stream so we can benchmark offload throughput using the production write path. Made-with: Cursor

Replace the hardcoded local offload stream with a generic factory registration point so benchmark runs can swap in Kafka, SQS, or another transport without rewriting the wrapper. Made-with: Cursor

Wire the SSHistoricalOffload benchmark backend to a simple Kafka/MSK producer so cryptosim can measure end-to-end offload throughput with a real queue while keeping the transport integration generic and small. Made-with: Cursor

Switch the cryptosim Kafka historical offload path to throughput-oriented defaults with async publishing, larger producer batches, and no required broker acks while keeping the settings configurable. Made-with: Cursor

Support AWS_MSK_IAM authentication for the cryptosim Kafka offload path so the benchmark can publish to MSK Serverless using normal AWS credential resolution while keeping the change localized to the Kafka transport. Made-with: Cursor

Allow the historical offload benchmark backend to skip transaction reads so Kafka offload runs exercise the write path without read-dependent execution failures. Made-with: Cursor

Route SSHistoricalOffload benchmark writes directly to the offload stream without persisting local state so the benchmark measures pure historical offload throughput. Made-with: Cursor

Align the rebased offload wrapper and tests with the current benchmark wrapper API so the branch stays buildable after landing on top of main. Made-with: Cursor

github-actions · 2026-04-07T15:21:45Z

The latest Buf updates on your PR. Results from workflow Buf / buf (pull_request).

Build	Format	Lint	Breaking	Updated (UTC)
`✅ passed`	`✅ passed`	`✅ passed`	`✅ passed`	Apr 8, 2026, 2:40 PM

Use the configured state store settings for the historical offload benchmark path and keep the wrapper state aligned with successful publishes. Made-with: Cursor

codecov · 2026-04-07T15:37:24Z

Codecov Report

❌ Patch coverage is 42.78075% with 107 lines in your changes missing coverage. Please review.
✅ Project coverage is 58.75%. Comparing base (6cb18cd) to head (2ee44ac).
⚠️ Report is 4 commits behind head on main.

Files with missing lines	Patch %	Lines
sei-db/state_db/ss/offload/aws_msk_iam.go	6.77%	55 Missing ⚠️
sei-db/state_db/ss/offload/kafka.go	59.37%	45 Missing and 7 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3183      +/-   ##
==========================================
- Coverage   58.95%   58.75%   -0.21%     
==========================================
  Files        2060     2057       -3     
  Lines      168673   168560     -113     
==========================================
- Hits        99444    99033     -411     
- Misses      60529    60777     +248     
- Partials     8700     8750      +50

Flag	Coverage Δ
sei-chain-pr	`42.78% <42.78%> (?)`
sei-db	`70.41% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
sei-db/state_db/ss/offload/kafka.go	`59.37% <59.37%> (ø)`
sei-db/state_db/ss/offload/aws_msk_iam.go	`6.77% <6.77%> (ø)`

... and 75 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Remove the unused state store replay and local offload hook so this branch only keeps the publish-only benchmark path and the Kafka transport it actually exercises. Made-with: Cursor

Deduplicate the repeated "none" option in the Kafka offload config so the targeted lint check passes without changing behavior. Made-with: Cursor

Remove the local buffered fallback and its tests so the historical offload benchmark path requires explicit Kafka configuration. Made-with: Cursor

sei-db/state_db/bench/cryptosim/historical_offload.go

Remove username/password-based Kafka auth paths so the benchmark config only keeps the auth modes used by the current offload flow. Made-with: Cursor

Provide a ready-to-edit cryptosim config for Kafka-based offchain pipeline testing so reviewers have a concrete example of the intended benchmark setup. Made-with: Cursor

Drop the indirect xdg-go dependencies that were left behind after removing the unused Kafka SCRAM auth path. Made-with: Cursor

sei-db/state_db/bench/cryptosim/config/historical-offload-kafka.json

sei-db/state_db/bench/cryptosim/cryptosim.go

sei-db/state_db/bench/wrappers/historical_offload_wrapper.go

sei-db/state_db/bench/cryptosim/cryptosim_config.go

Move Kafka stream construction into the historical offload wrapper path, remove the global factory setup, and trim the sample config down to the non-default settings reviewers need to edit. Made-with: Cursor

* main: Add receipt / log reads to cryptosim (#3081) persist blocks and FullCommitQCs in data layer via WAL (CON-231) (#3126) Update Changelog in prep to cut v6.4.1 (#3213) fix(sei-tendermint): resolve staticcheck warnings (#3207) Add historical state offload stream hook (#3183) feat: wire autobahn config propagation from top-level to GigaRouter (CON-232) (#3194)

Kbhat1 added 9 commits April 7, 2026 11:14

add historical state offload stream hook

bdb6c96

Introduce a transport-agnostic changelog stream interface so historical state can be published to an external queue and replayed later without coupling the state store to a specific message system. Made-with: Cursor

add cryptosim historical offload backend

66db465

Add a write-only cryptosim backend that exercises the SS historical offload hook with a buffered in-process stream so we can benchmark offload throughput using the production write path. Made-with: Cursor

make cryptosim offload transport pluggable

2ab10fc

Replace the hardcoded local offload stream with a generic factory registration point so benchmark runs can swap in Kafka, SQS, or another transport without rewriting the wrapper. Made-with: Cursor

add cryptosim kafka historical offload transport

13334b4

Wire the SSHistoricalOffload benchmark backend to a simple Kafka/MSK producer so cryptosim can measure end-to-end offload throughput with a real queue while keeping the transport integration generic and small. Made-with: Cursor

tune cryptosim kafka offload for throughput

2377556

Switch the cryptosim Kafka historical offload path to throughput-oriented defaults with async publishing, larger producer batches, and no required broker acks while keeping the settings configurable. Made-with: Cursor

add msk iam auth for cryptosim kafka offload

986360e

Support AWS_MSK_IAM authentication for the cryptosim Kafka offload path so the benchmark can publish to MSK Serverless using normal AWS credential resolution while keeping the change localized to the Kafka transport. Made-with: Cursor

bench: add write-only mode to historical offload cryptosim

290a294

Allow the historical offload benchmark backend to skip transaction reads so Kafka offload runs exercise the write path without read-dependent execution failures. Made-with: Cursor

bench: make historical offload wrapper publish only

8385473

Route SSHistoricalOffload benchmark writes directly to the offload stream without persisting local state so the benchmark measures pure historical offload throughput. Made-with: Cursor

fix historical offload rebase fallout

8cc7ebc

Align the rebased offload wrapper and tests with the current benchmark wrapper API so the branch stays buildable after landing on top of main. Made-with: Cursor

Kbhat1 force-pushed the historical-state-offload-flow branch from cd91224 to 8cc7ebc Compare April 7, 2026 15:20

clean up historical offload benchmark config wiring

959a2b8

Use the configured state store settings for the historical offload benchmark path and keep the wrapper state aligned with successful publishes. Made-with: Cursor

Kbhat1 added the non-app-hash-breaking label Apr 7, 2026

Kbhat1 added 4 commits April 7, 2026 14:13

Merge branch 'main' into historical-state-offload-flow

cbddcc8

trim historical offload to the benchmark write path

bcc8b82

Remove the unused state store replay and local offload hook so this branch only keeps the publish-only benchmark path and the Kafka transport it actually exercises. Made-with: Cursor

fix kafka offload lint constant

d23d401

Deduplicate the repeated "none" option in the Kafka offload config so the targeted lint check passes without changing behavior. Made-with: Cursor

make historical offload benchmark kafka-only

049d538

Remove the local buffered fallback and its tests so the historical offload benchmark path requires explicit Kafka configuration. Made-with: Cursor

Kbhat1 requested review from cody-littley, jewei1997 and yzang2019 April 7, 2026 19:05

yzang2019 reviewed Apr 7, 2026

View reviewed changes

sei-db/state_db/bench/cryptosim/historical_offload.go Outdated Show resolved Hide resolved

Kbhat1 added 2 commits April 7, 2026 15:42

trim unused kafka auth options from historical offload

c8c5260

Remove username/password-based Kafka auth paths so the benchmark config only keeps the auth modes used by the current offload flow. Made-with: Cursor

add sample config for historical offload testing

f376c3a

Provide a ready-to-edit cryptosim config for Kafka-based offchain pipeline testing so reviewers have a concrete example of the intended benchmark setup. Made-with: Cursor

yzang2019 approved these changes Apr 7, 2026

View reviewed changes

Kbhat1 marked this pull request as ready for review April 7, 2026 20:00

remove stale scram module dependencies

51b3adc

Drop the indirect xdg-go dependencies that were left behind after removing the unused Kafka SCRAM auth path. Made-with: Cursor

cody-littley reviewed Apr 7, 2026

View reviewed changes

clean up historical offload benchmark wiring

8434d8e

Move Kafka stream construction into the historical offload wrapper path, remove the global factory setup, and trim the sample config down to the non-default settings reviewers need to edit. Made-with: Cursor

Kbhat1 requested a review from cody-littley April 8, 2026 13:44

Kbhat1 added 2 commits April 8, 2026 09:47

Merge branch 'main' into historical-state-offload-flow

627c340

Merge branch 'main' into historical-state-offload-flow

2ee44ac

cody-littley approved these changes Apr 8, 2026

View reviewed changes

Kbhat1 added this pull request to the merge queue Apr 8, 2026

Merged via the queue into main with commit 47afb6e Apr 8, 2026
39 checks passed

Kbhat1 deleted the historical-state-offload-flow branch April 8, 2026 19:19

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add historical state offload stream hook#3183

Add historical state offload stream hook#3183
Kbhat1 merged 20 commits intomainfrom
historical-state-offload-flow

Kbhat1 commented Apr 3, 2026 •

edited

Loading

Uh oh!

github-actions bot commented Apr 7, 2026 •

edited

Loading

Uh oh!

codecov bot commented Apr 7, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

Kbhat1 commented Apr 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Describe your changes and provide context

Testing performed to validate your change

Uh oh!

github-actions bot commented Apr 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Apr 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Kbhat1 commented Apr 3, 2026 •

edited

Loading

github-actions bot commented Apr 7, 2026 •

edited

Loading

codecov bot commented Apr 7, 2026 •

edited

Loading