Skip to content

Fix rollback failure due to snapshot creation happened after app hash#2680

Merged
yzang2019 merged 3 commits intomainfrom
yzang/rollback-failure
Jan 9, 2026
Merged

Fix rollback failure due to snapshot creation happened after app hash#2680
yzang2019 merged 3 commits intomainfrom
yzang/rollback-failure

Conversation

@yzang2019
Copy link
Copy Markdown
Contributor

Describe your changes and provide context

Problem:
Currently when a node app hashed, and the node was running in that state for a long enough time, snapshot creation will kick in on latest block height. And if the node was configured to only store 1 snapshot, then the old snapshot height will be truncated. That will lead to rollback failure because there's no way to rollback to a height that is older than the only snapshot exist.

Mitigation:
Previously we made a change to be able to create snapshot at any given height, as long as the height difference is > interval. This PR revert that changes so that snapshot creation will only happen if the block height % snapshot interval == 0. This should greatly reduce the chance of rollback failure leading to snapshot being created on latest height.

Testing performed to validate your change

@github-actions
Copy link
Copy Markdown

github-actions bot commented Jan 8, 2026

The latest Buf updates on your PR. Results from workflow Buf / buf (pull_request).

BuildFormatLintBreakingUpdated (UTC)
✅ passed✅ passed✅ passed✅ passedJan 8, 2026, 10:15 PM

@codecov
Copy link
Copy Markdown

codecov bot commented Jan 8, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 43.83%. Comparing base (426055b) to head (5160602).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #2680      +/-   ##
==========================================
+ Coverage   43.82%   43.83%   +0.01%     
==========================================
  Files        1908     1908              
  Lines      159073   159073              
==========================================
+ Hits        69708    69731      +23     
+ Misses      82958    82939      -19     
+ Partials     6407     6403       -4     
Flag Coverage Δ
sei-chain 45.89% <100.00%> (+<0.01%) ⬆️
sei-cosmos 38.21% <ø> (+<0.01%) ⬆️
sei-db 69.13% <ø> (ø)
sei-tendermint 47.35% <ø> (+0.04%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
sei-db/state_db/sc/memiavl/db.go 63.14% <100.00%> (ø)

... and 11 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@yzang2019 yzang2019 enabled auto-merge (squash) January 9, 2026 03:55
@yzang2019 yzang2019 merged commit 760e3ca into main Jan 9, 2026
39 checks passed
@yzang2019 yzang2019 deleted the yzang/rollback-failure branch January 9, 2026 10:13
yzang2019 added a commit that referenced this pull request Feb 25, 2026
…#2680)

## Describe your changes and provide context
Problem:
Currently when a node app hashed, and the node was running in that state
for a long enough time, snapshot creation will kick in on latest block
height. And if the node was configured to only store 1 snapshot, then
the old snapshot height will be truncated. That will lead to rollback
failure because there's no way to rollback to a height that is older
than the only snapshot exist.

Mitigation:
Previously we made a change to be able to create snapshot at any given
height, as long as the height difference is > interval. This PR revert
that changes so that snapshot creation will only happen if the block
height % snapshot interval == 0. This should greatly reduce the chance
of rollback failure leading to snapshot being created on latest height.

## Testing performed to validate your change
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants