Skip to content

fix: accept legacy CommitHash and prevent mmap use-after-free for v6.3->v6.4 upgrade#3095

Closed
masih wants to merge 2 commits intorelease/v6.4from
cursor/last-commit-hash-error-9a3f
Closed

fix: accept legacy CommitHash and prevent mmap use-after-free for v6.3->v6.4 upgrade#3095
masih wants to merge 2 commits intorelease/v6.4from
cursor/last-commit-hash-error-9a3f

Conversation

@masih
Copy link
Copy Markdown
Collaborator

@masih masih commented Mar 20, 2026

Describe your changes and provide context

Two fixes for deploying release/v6.4 on nodes previously running release/v6.3:

Fix 1: LastCommitHash mismatch panic on startup

The Commit.Hash() algorithm was changed in CON-76 (#2600) to include Height, Round, and BlockID in the Merkle tree alongside signatures. Blocks stored by v6.3 nodes have LastCommitHash computed with the old signatures-only algorithm. When v6.4 loads these blocks during ABCI handshake replay, ValidateBasic recomputes the hash with the new algorithm, producing a different value and causing a panic:

panic: error from proto block: wrong Header.LastCommitHash. Expected 75AA2EF6..., got E3CCCADC...

Adds Commit.LegacyHash() that uses the old (signatures-only) algorithm, and updates Block.ValidateBasic() to fall back to it when the new hash doesn't match.

Fix 2: SIGSEGV in historical RPC queries (use-after-free of mmap'd memory)

When serving historical queries, rootmulti.Store.Query() opens a read-only memiavl instance (mmap'd), queries it, and defers Close(). The response byte slices (Key, Value) may point directly into the mmap'd region. When the deferred Close() unmaps the memory before JSON marshaling reads the response, the process segfaults:

fatal error: fault
[signal SIGSEGV: segmentation violation code=0x1 addr=0x7fd224098d0a]
encoding/base64.(*Encoding).Encode(...)

Copies res.Key and res.Value for non-latest queries so they survive the mmap unmap.

Note on PR #3093

The IBC Upgrade Params fix from #3093 is already included in this branch since it was merged into release/v6.4 before this branch was created.

Testing performed to validate your change

  • go test ./sei-tendermint/types/ — all tests pass including TestCommitHash and TestBlockValidateBasic
  • go test ./sei-tendermint/internal/store/ — all store tests pass
  • Verified that tampered commits are still correctly rejected (neither new nor legacy hash matches)
  • Fix 1 confirmed working on arctic-1 RPC node (handshake completed successfully)
Open in Web Open in Cursor 

…grade

The Commit.Hash() algorithm changed in CON-76 (#2600) to include
Height, Round, and BlockID in the Merkle tree alongside signatures.
Blocks stored by v6.3 nodes have LastCommitHash computed with the old
signatures-only algorithm. When v6.4 loads these blocks, ValidateBasic
recomputes the hash with the new algorithm, causing a mismatch panic.

Add Commit.LegacyHash() that uses the old algorithm, and fall back to
it in ValidateBasic when the new hash doesn't match.

Co-authored-by: Masih H. Derkani <m@derkani.org>
@github-actions
Copy link
Copy Markdown

github-actions bot commented Mar 20, 2026

The latest Buf updates on your PR. Results from workflow Buf / buf (pull_request).

BuildFormatLintBreakingUpdated (UTC)
✅ passed✅ passed✅ passed✅ passedMar 20, 2026, 12:10 PM

@codecov
Copy link
Copy Markdown

codecov bot commented Mar 20, 2026

Codecov Report

❌ Patch coverage is 77.77778% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 58.38%. Comparing base (567ec92) to head (111c0d3).
⚠️ Report is 2 commits behind head on release/v6.4.

Files with missing lines Patch % Lines
sei-tendermint/types/block.go 69.23% 2 Missing and 2 partials ⚠️
Additional details and impacted files

Impacted file tree graph

@@              Coverage Diff              @@
##           release/v6.4    #3095   +/-   ##
=============================================
  Coverage         58.38%   58.38%           
=============================================
  Files              2088     2088           
  Lines            172082   172099   +17     
=============================================
+ Hits             100466   100481   +15     
  Misses            62659    62659           
- Partials           8957     8959    +2     
Flag Coverage Δ
sei-chain-pr 77.00% <77.77%> (?)
sei-db 70.41% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
sei-cosmos/storev2/rootmulti/store.go 46.97% <100.00%> (+0.47%) ⬆️
sei-tendermint/types/block.go 87.31% <69.23%> (-0.08%) ⬇️
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

…-after-free

When serving historical queries, the Query method opens a read-only
memiavl instance (mmap'd), queries it, and defers Close(). The response
byte slices (Key, Value) may point directly into the mmap'd region.
When the deferred Close() unmaps the memory before JSON marshaling
reads the response, the process segfaults (SIGSEGV).

Copy res.Key and res.Value for non-latest queries so they survive
the mmap unmap.

Co-authored-by: Masih H. Derkani <m@derkani.org>
@cursor cursor bot changed the title fix: accept legacy CommitHash in ValidateBasic to allow v6.3->v6.4 upgrade fix: accept legacy CommitHash and prevent mmap use-after-free for v6.3->v6.4 upgrade Mar 20, 2026
github-merge-queue bot pushed a commit that referenced this pull request Mar 20, 2026
The Commit.Hash() algorithm was changed in CON-76 (#2600) to include
Height, Round, and BlockID in the Merkle tree alongside signatures.
Blocks stored by v6.3 nodes have LastCommitHash computed with the old
signatures-only algorithm. When v6.4 loads these blocks during ABCI
handshake replay, ValidateBasic recomputes the hash with the new
algorithm, producing a different value and causing a panic first
observed on `arctic-1`.

Adds Commit.LegacyHash() that uses the old (signatures-only) algorithm,
and updates Block.ValidateBasic() to fall back to it when the new hash
doesn't match.

Note that release 6.5 would make this fallback path entirely redundant,
and it should be removed in the next release.

Separated out from #3095
to capture a clear commit history of changes for the future
archeologists.
github-actions bot pushed a commit that referenced this pull request Mar 20, 2026
The Commit.Hash() algorithm was changed in CON-76 (#2600) to include
Height, Round, and BlockID in the Merkle tree alongside signatures.
Blocks stored by v6.3 nodes have LastCommitHash computed with the old
signatures-only algorithm. When v6.4 loads these blocks during ABCI
handshake replay, ValidateBasic recomputes the hash with the new
algorithm, producing a different value and causing a panic first
observed on `arctic-1`.

Adds Commit.LegacyHash() that uses the old (signatures-only) algorithm,
and updates Block.ValidateBasic() to fall back to it when the new hash
doesn't match.

Note that release 6.5 would make this fallback path entirely redundant,
and it should be removed in the next release.

Separated out from #3095
to capture a clear commit history of changes for the future
archeologists.

(cherry picked from commit 512976d)
github-merge-queue bot pushed a commit that referenced this pull request Mar 20, 2026
When serving historical queries, rootmulti.Store.Query() opens a
read-only memiavl instance (mmap'd), queries it, and defers Close(). The
response byte slices (Key, Value) may point directly into the mmap'd
region. When the deferred Close() unmaps the memory before JSON
marshaling reads the response, the process segfaults (observed on
`arctic-1`.

Copies res.Key and res.Value for non-latest queries so they survive the
mmap unmap.

Separated out from #3095
to capture a clear commit history of changes for the future
archeologists.
github-actions bot pushed a commit that referenced this pull request Mar 20, 2026
When serving historical queries, rootmulti.Store.Query() opens a
read-only memiavl instance (mmap'd), queries it, and defers Close(). The
response byte slices (Key, Value) may point directly into the mmap'd
region. When the deferred Close() unmaps the memory before JSON
marshaling reads the response, the process segfaults (observed on
`arctic-1`.

Copies res.Key and res.Value for non-latest queries so they survive the
mmap unmap.

Separated out from #3095
to capture a clear commit history of changes for the future
archeologists.

(cherry picked from commit 145f854)
@jsand11
Copy link
Copy Markdown

jsand11 commented Mar 20, 2026

// File: sei-db/common/safe.go (recommended location)
// Package common provides shared utility functions used across sei-db modules.

package common

import "github.com/cosmos/cosmos-sdk/store/types"

// SafeCopySlice creates a new independent copy of the input byte slice.
// Returns nil if the input is nil.
//
// This function should be used when dealing with byte slices that originate
// from mmap'd memory regions (e.g. MemIAVL / state store queries) to prevent
// use-after-free bugs when the underlying memory is unmapped (e.g. after
// defer store.Close()).
//
// Example usage in Query methods:
//
// res.Key = common.SafeCopySlice(res.Key)
// res.Value = common.SafeCopySlice(res.Value)
func SafeCopySlice(b []byte) []byte {
if b == nil {
return nil
}
c := make([]byte, len(b))
copy(c, b)
return c
}

// SafeCopyStoreValue is a convenience wrapper that copies both Key and Value
// fields of a types.StoreKVPair (used in some iterator / proof paths).
func SafeCopyStoreValue(kv types.StoreKVPair) types.StoreKVPair {
return types.StoreKVPair{
Key: SafeCopySlice(kv.Key),
Value: SafeCopySlice(kv.Value),
}
}

// SafeCopyBytes is an alias for SafeCopySlice — use whichever name feels
// more readable in context.
func SafeCopyBytes(b []byte) []byte {
return SafeCopySlice(b)
}

@masih masih closed this Mar 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants