Database gets corrupted on --init.reorg

[nuliknol]

I missed the fork again (had no time to check for updates, but you released incompatible change), my database got corrupted again. After trying to "setHead()" backwards and startup with pre-fork state , the database got corrupted again , here is the error log that I get:

15 messageCount=220928079 l1Block=9,748,574 l1Timestamp=2025-12-01T22:00:23+0000
WARN [12-01|22:08:20.651] error reading inbox                      err="failed to get previous message for msgIdx 220926634: pebble: not found"
INFO [12-01|22:08:21.811] InboxTracker                             sequencerBatchCount=628,115 messageCount=220928079 l1Block=9,748,574 l1Timestamp=2025-12-01T22:00:23+0000
WARN [12-01|22:08:21.812] error reading inbox                      err="failed to get previous message for msgIdx 220926634: pebble: not found"
INFO [12-01|22:08:22.969] InboxTracker                             sequencerBatchCount=628,115 messageCount=220928079 l1Block=9,748,574 l1Timestamp=2025-12-01T22:00:23+0000
WARN [12-01|22:08:22.969] error reading inbox                      err="failed to get previous message for msgIdx 220926634: pebble: not found"
INFO [12-01|22:08:24.144] InboxTracker                             sequencerBatchCount=628,115 messageCount=220928079 l1Block=9,748,574 l1Timestamp=2025-12-01T22:00:23+0000
WARN [12-01|22:08:24.144] error reading inbox                      err="failed to get previous message for msgIdx 220926634: pebble: not found"
INFO [12-01|22:08:25.304] InboxTracker                             sequencerBatchCount=628,115 messageCount=220928079 l1Block=9,748,574 l1Timestamp=2025-12-01T22:00:23+0000
WARN [12-01|22:08:25.304] error reading inbox                      err="failed to get previous message for msgIdx 220926634: pebble: not found"
INFO [12-01|22:08:26.449] InboxTracker                             sequencerBatchCount=628,115 messageCount=220928079 l1Block=9,748,574 l1Timestamp=2025-12-01T22:00:23+0000
WARN [12-01|22:08:26.449] error reading inbox                      err="failed to get previous message for msgIdx 220926634: pebble: not found"
INFO [12-01|22:08:27.595] InboxTracker                             sequencerBatchCount=628,115 messageCount=220928079 l1Block=9,748,574 l1Timestamp=2025-12-01T22:00:23+0000
WARN [12-01|22:08:27.595] error reading inbox                      err="failed to get previous message for msgIdx 220926634: pebble: not found"
INFO [12-01|22:08:28.748] InboxTracker                             sequencerBatchCount=628,115 messageCount=220928079 l1Block=9,748,574 l1Timestamp=2025-12-01T22:00:23+0000
WARN [12-01|22:08:28.749] error reading inbox                      err="failed to get previous message for msgIdx 220926634: pebble: not found"
INFO [12-01|22:08:29.890] InboxTracker                             sequencerBatchCount=628,115 messageCount=220928079 l1Block=9,748,574 l1Timestamp=2025-12-01T22:00:23+0000
WARN [12-01|22:08:29.890] error reading inbox                      err="failed to get previous message for msgIdx 220926634: pebble: not found"
INFO [12-01|22:08:31.052] InboxTracker                             sequencerBatchCount=628,115 messageCount=220928079 l1Block=9,748,574 l1Timestamp=2025-12-01T22:00:23+0000
WARN [12-01|22:08:31.053] error reading inbox                      err="failed to get previous message for msgIdx 220926634: pebble: not found"

I ran nitro with this extra parameter: --init.reorg-to-block-batch 220860000

The question is, why are you allowing user's command to get the DB into invalid state? User validation is like the basic requirement any piece of software should comply with. Now I have to rebuild the entire database and it takes days

[nuliknol]

I was able to recover my DB by running --init.reorg-to-block-batch set to block 1 number less than 220863979 (the block at which ArbOS version 51 was enabled) after I discovered that this time, the chain didn't went on its own, creating a new fork, but stopped at block 220863979 (in previous fork it went its own way with new block hashes different form the main chain).

But anyway, --init.reorg-to-block-batch should not break things if going to much into the past

[joshuacolvin0]

You can only reorg archive nodes to any block, full nodes don't save any extra information.
Any parameter not explicitly documented on this page should only be used if you fully understand the risk:
https://docs.arbitrum.io/run-arbitrum-node/run-full-node

[gligneul]

@nuliknol, if I understood correctly, you were able to fix your node by using --init.reorg-to-block-batch, correct? As Josh pointed out, this is an advanced feature.

[nuliknol]

@nuliknol, if I understood correctly, you were able to fix your node by using --init.reorg-to-block-batch, correct? As Josh pointed out, this is an advanced feature.

yep, I was able to. Probably because I rolled back just 1 block backwards. And because the state is kept for the last 128 states, I can do that. (but only 128 blocks bac). Because the Node stopped creating new blocks, and just stalled with continuous error (no new blocks were created), I was able to do the "debug.setHead()" with --init.reorg-to-block-batch option

Therefore, the validation you should add to the code is:

does the state of block to roll back exists? 
if (it doesnt) then { show error }
else { do the debug.setHead() }

@gligneul

[rdin777]

"I've been reviewing the logs you shared. The 'failed to get previous message' error during --init.reorg suggests a potential inconsistency in the message index mapping within the database after the setHead() operation.

To help diagnose if this is a software bug or an infrastructure bottleneck, could you provide more details:

Node Version: Are you running v3.9.3 or a different version?

Storage Setup: What kind of drive is the database stored on (e.g., NVMe, SATA SSD)?

Hardware Specs: How much RAM is allocated to the process during the reorg?

Log Context: Did you notice any 'out of memory' (OOM) errors or disk I/O spikes just before the database corruption occurred?

I've seen similar patterns where high disk latency during head resets can lead to partial state writes."

[joshuacolvin0]

reorg-to-block-batch is not meant ot be user-friendly, adding in extra checks would limit flexibility