Improved scanner modes

[syntrust]

Addressing #483, this PR implemented more scanner modes for more choices. Check the doc for more details.

Tested hybrid scan modes on testnet with config:

--scanner.mode 7 \
--scanner.batch-size 1000 \
--scanner.interval.meta 10 \
--scanner.interval.blob 60 \
--scanner.interval.block 1440 \

INFO [01-06|12:08:59.534] Scanner update loop received event - all shards done.
INFO [01-06|12:08:59.534] Scanner started                          mode=check-meta+check-blob+check-block
INFO [01-06|12:08:59.534] Launching scanner loop                   mode=check-meta                        interval=10m0s batchSize=1000
INFO [01-06|12:08:59.534] Launching scanner loop                   mode=check-block                       interval=24h0m0s batchSize=7200
INFO [01-06|12:08:59.534] Local storage summary                    localKvs=shard0[0-17249] localKvCount=17250
INFO [01-06|12:08:59.534] Launching scan fix loop                  interval=12m0s
INFO [01-06|12:08:59.534] Launching scanner loop                   mode=check-blob                        interval=1h0m0s  batchSize=1000
INFO [01-06|12:08:59.534] Scan batch started                       mode=check-meta                        startIndexOfKvIdx=0 kvsInBatch=[0-999]
INFO [01-06|12:08:59.560] Scan batch done                          mode=check-meta                        scanned=[0-999] count=1000 nextIndexOfKvIdx=1000 duration=26.427169ms
INFO [01-06|12:08:59.561] No last scanned block recorded, starting from 7200 slots ago
INFO [01-06|12:08:59.615] Latest updated KV indices fetched        startBlock=9,982,549 endBlock=9,989,749 totalUpdatedKvs=18 locallyStored=18
INFO [01-06|12:08:59.615] Scan batch started                       mode=check-block                       startIndexOfKvIdx=0 kvsInBatch=[17232-17249]
INFO [01-06|12:09:02.067] Scan batch done                          mode=check-block                       scanned=[17232-17249] count=18   nextIndexOfKvIdx=9,989,749 duration=2.507269601s
INFO [01-06|12:09:02.067] Scan batch started                       mode=check-blob                        startIndexOfKvIdx=0 kvsInBatch=[0-999]
INFO [01-06|12:09:49.486] Scan batch done                          mode=check-blob                        scanned=[0-999]       count=1000 nextIndexOfKvIdx=1000      duration=47.419013179s
INFO [01-06|12:09:59.535] Scanner stats                            mode=check-meta+check-blob+check-block mismatched=(none)
INFO [01-06|12:20:59.535] Scanner fix batch triggered
...
INFO [01-07|12:08:59.535] Local storage summary                    localKvs=shard0[0-17260] localKvCount=17261
INFO [01-07|12:08:59.535] Scanner stats                            mode=check-meta+check-blob+check-block mismatched=(none)
INFO [01-07|12:08:59.625] Latest updated KV indices fetched        startBlock=9,989,750 endBlock=9,996,058 totalUpdatedKvs=12 locallyStored=12
INFO [01-07|12:08:59.626] Scan batch started                       mode=check-block                       startIndexOfKvIdx=9,989,749 kvsInBatch=[17250-17254],[472],[17255-17260]
INFO [01-07|12:09:00.186] Scan batch done                          mode=check-block                       scanned=[17250-17254],[472],[17255-17260] count=12   nextIndexOfKvIdx=9,996,058 duration=652.352155ms
INFO [01-07|12:09:00.186] Scan batch started                       mode=check-blob                        startIndexOfKvIdx=5000      kvsInBatch=[5000-5999]
INFO [01-07|12:09:47.915] Scan batch done                          mode=check-blob                        scanned=[5000-5999]                       count=1000 nextIndexOfKvIdx=6000      duration=47.728831812s
INFO [01-07|12:09:47.915] Scan batch started                       mode=check-meta                        startIndexOfKvIdx=17000     kvsInBatch=[17000-17260]
INFO [01-07|12:09:47.927] Scan batch done                          mode=check-meta                        scanned=[17000-17260]                     count=261  nextIndexOfKvIdx=17261     duration=11.833072ms

Tested blob fix in local devnet:

INFO [01-13|18:20:58.393] Scan batch started                       mode=check-blob startIndexOfKvIdx=0 kvsInBatch=[160-207],[216-223]
WARN [01-13|18:21:03.647] Commit mismatch detected                 kvIndex=223 error="commit mismatch: expected=011ee3ee378e9349d1e239023f2c9bdb50b6e9648a78ea32, found=011d0a55a70937e244d598943e961ed2cf449d4d799ffb4d"
INFO [01-13|18:32:58.379] Scanner stats                            mode=check-blob mismatched=[223(pending)]
INFO [01-13|18:32:58.379] Scanner fix batch triggered
INFO [01-13|18:32:58.379] Scanner fixing batch                     mismatches=[223]
INFO [01-13|18:32:58.798] Fixing mismatched KV                     kvIndex=223
INFO [01-13|18:32:59.198] KV fixed successfully                    kvIndex=223
INFO [01-13|18:33:58.376] Scanner stats                            mode=check-blob mismatched=[]

[qzhodl]

This is a pretty large PR and it’s hard to review as a single chunk. Could we split it into a few smaller PRs?

For example:
PR 1 (optional): Pure refactoring/cleanup (no functional changes), if needed.
PR 2: Add support for hybrid scan modes on the existing scan modes (no behavior change beyond the hybrid support).
PR 3: Add block mode support.

This would make the review cycle much faster and reduce the potential bugs.