feat: add masternode network generator with real DIP-0024 quorums#4
Conversation
Multi-node masternode network generator for regtest test data. Produces 5 datadirs (1 controller + 4 MNs) with DKG cycles and quorum history for SPV masternode list sync integration testing. Follows Dash Core test framework ordering: - mocktime init -> force mnsync -> connect nodes - setmnthreadactive toggle during peer connections - DKG phase-by-phase progression with mockscheduler
Remove the -llmqtestinstantsenddip0024=llmq_test_instantsend override that was switching the DIP0024 InstantSend quorum type from LLMQ_TEST_DIP0024 (useRotation=true) to LLMQ_TEST_INSTANTSEND (useRotation=false). This prevented quorum snapshots from being created during block processing, causing BuildQuorumRotationInfo to fail with "Cannot find quorum snapshot". Also use RPC stop for clean node shutdown to flush evoDB.
Rework `phase_5_mine_dkg_cycles` to follow Dash Core's `mine_cycle_quorum` flow from `test/functional/test_framework/test_framework.py`. The previous generator drove only a single `llmq_test` (type 100) session per cycle and never orchestrated the interleaved `llmq_test_dip0024` (type 103) rotating DKG, so every DIP-0024 commitment landed as null and the SPV consumer tests saw zeroed `signers`, `validMembers`, and `quorumPublicKey` for type 103. Each cycle now: - advances 3 full DKG intervals before the first run (`cycle_quorum_is_ready` warmup) so rotating quorums can form past `H+3C`, matching `feature_llmq_rotation.py` - walks phases 1-6 block-by-block, interleaving `q_0` and `q_1` for type 103 while type 100 piggybacks naturally on the shared `dkgInterval=24` - gates every phase on the expected DKG message counts (`receivedContributions`, `receivedComplaints`, `receivedJustifications`, `receivedPrematureCommitments`) so blocks never advance past a phase before real messages have been exchanged - mines a single commit block at `cycle+12` via `getblocktemplate + generate(1)`, matching `mine_cycle_quorum`'s terminal step - verifies all three real quorums via `quorum list` (no `count` arg, so the default returns both `q_0` and `q_1` for rotating types — the prior `count=1` dropped `q_1` from the response even when it was successfully mined) - mines 8 signing-window maturity blocks Also seeds direct masternode↔masternode `addnode` connections in `MasternodeNetwork.connect_all` so DKG phase 2 sees real contributions instead of racing against the quorum manager's lazy connection build, and switches node shutdown to RPC `stop` with a SIGTERM fallback so evoDB and quorum snapshots are flushed cleanly.
Consolidate scattered lifecycle helpers and remove dead plumbing uncovered while fixing the DIP-0024 cycle generator. - Move mocktime control onto `MasternodeNetwork` as `set_mocktime` / `bump_mocktime` / `move_blocks`, plus `all_nodes()`. The previous free functions poked at `network._mocktime` from outside the class and the `hasattr(self, "_mocktime")` guards in `start_masternode_nodes` were paranoid dead checks since `_mocktime` is always set before any node starts. - Move `force_finish_mnsync` and mocktime application onto `MasternodeNode`. - Hoist `import subprocess` to the module top (per the project's import-at-top rule). - Drop the unreachable `find_free_port(controller_p2p_port + ...)` fallback in `start_masternode_nodes`: `allocate_mn_ports()` always runs first, so `mn_p2p_ports` is always populated. Replace with a direct `assert`. - Replace the `MasternodeConfig` dataclass with direct use of argparse `args`. It was a three-field indirection used exactly once. - Remove the `completed_cycles` counter and the `dkg_cycles` pass-through from `phase_5_mine_dkg_cycles` to `phase_7_export`. Since `_run_single_dkg_cycle` raises on failure, the counter always equalled `num_cycles`. - Make `wait_for_quorum_phase`, `wait_for_quorum_connections`, and `wait_for_quorum_list` raise `DKGCycleError` with a diagnostic dump on timeout. This eliminates ~12 call sites of 6-line `_require(...)` boilerplate and pushes the context into the helper that has it. - Extract `_find_session`, `_find_connection_group`, `_wait_for_dkg_phase`, and `PHASE_GATES` / `PHASE_NAMES` / `LLMQ_TYPE_NUM` tables to collapse duplicated dkgstatus navigation and the per-phase q_0/q_1 interleaving. - Extract a small `try_addnode` closure in `MasternodeNetwork.connect_all` to replace the four copy-pasted `try/except` addnode blocks. Net effect: `generate_masternode.py` shrinks from 884 to ~690 lines and `masternode_network.py` from 363 to ~400 lines (extra methods offset by removed boilerplate). Verified end-to-end with `--dkg-cycles 2`: real `llmq_test` and `llmq_test_dip0024` quorums formed in every cycle.
Add `phase_6b_extend_to_quiet_tip` that stops the masternodes and mines controller-only blocks past the next cycle's mining window (+8 maturity) before export. For the default 8-cycle run this lands the exported tip at height 460, matching the formula `current_cycle_start + DKG_INTERVAL + dkgMiningWindowEnd + SIGN_HEIGHT_OFFSET`. Previously, phase 6 left the chain tip inside a DKG cycle that had started but not reached its mining window (tip 412, inside cycle 408's phase 3). Test harnesses bringing the masternodes back online at that tip would miss phases 1-3 of the in-progress cycle, fail to contribute, and produce null commitments when their mine_dkg_cycle extended past the mining window. That left a future QRInfo diff with no ChainLock sigs for the rotation cycle, tripping the rust-dashcore QRInfo pre-check "Missing rotation ChainLock signatures in QRInfo: sigm0" in `test_instantsend_islock_arrives_before_tx`. Stopping the masternodes before extending guarantees the intermediate cycles settle as null commitments (no MN to broadcast `qfcommit`, so the controller's local `minableCommitmentsByQuorum` is empty at mining time and null commits land deterministically). By the exported tip the mining window has closed, so null commits are already written to evoDB and the cycle is no longer "in progress". When the test harness later restarts the masternodes, it sees a settled chain and can drive a fresh DKG for the next cycle boundary via `mine_dkg_cycle` without racing a half-lived DKG. Extract `DKG_MINING_WINDOW_END` and `SIGN_HEIGHT_OFFSET` as module constants and use them in phase 5's maturity mine (replacing the inline `8`) so the window arithmetic lives in one place.
The previous `phase_6b_extend_to_quiet_tip` (added in 4ec740f) shut down masternodes and mined controller-only blocks past the next cycle's mining window to "pre-settle" in-progress DKGs as null commits. That inverts the actual requirement: null rotating commits in the exported chain are precisely what trips rust-dashcore's QRInfo pre-check. The concrete failure mode, traced against Dash Core and rust-dashcore: - When the SPV's QRInfo diff covers a rotating cycle whose commitment is null, Dash Core's `BuildQuorumChainlockInfo` walks `new_quorums` and calls `qman.GetQuorum(type, quorumHash)`. Failed DKG commits have no quorum object, so that call returns null and the diff arrives with the null cycle's slot empty in `quorumsCLSigs`. - rust-dashcore's `apply_diff(mn_list_diff_h)` then cannot pull `sigm0` / `sigm1` / ... for that rotating slot, so `maybe_sigmN = None` and `feed_qr_info` rejects the QRInfo with `Missing rotation ChainLock signatures in QRInfo: sigmN`. The correct pattern, matching upstream's `mine_quorum` + `move_blocks` convention: - Keep all masternodes running for the whole chain-gen. - After the last orchestrated DKG cycle (phase 5 leaves tip at `cycle_N + dkgMiningWindowEnd` = `cycle_N + 20`), send the SPV test transactions and mine just enough blocks to confirm them AND to land the final tip in the DKG Idle gap at `cycle_N + 23`. - `cycle_N + 23` is the last Idle-phase block before the next cycle's phase 1 at `cycle_N + 24`: no DKG is mid-flight at the exported tip, every rotating commit on-chain is real, and a subsequent QRInfo's work block at `tip - 8 = 399` (for N=384) already sees the latest cycle's commit at `cycle_N + 12 = 396`. Changes: - Remove `phase_6b_extend_to_quiet_tip` entirely and drop its call from `main`. - Rework `phase_6_generate_test_transactions` to send all SPV transactions into the mempool, then mine exactly `DKG_INTERVAL - 1 - DKG_MINING_WINDOW_END` = 3 blocks — lands tip at `cycle_N + 23` and confirms every transaction with ≥1 confirmation. Add an assertion that the entry tip is exactly `cycle_N + DKG_MINING_WINDOW_END` and that the final tip's cycle offset lands in the Idle gap `(DKG_MINING_WINDOW_END, DKG_INTERVAL)`. Verified end-to-end: tip=407 (407 % 24 = 23 ∈ [21, 23]), 8 real `llmq_test` quorums and 16 real `llmq_test_dip0024` rotating quorums (all with non-zero `quorumPublicKey`), zero null commits anywhere in the chain.
Previously the exit tip landed at `cycle_N + 23` — the last block of the Idle gap, flush against the next cycle's phase 1 at `cycle_N + 24`. Any test harness that mines a single block during its setup before querying the chain (a common pattern for advancing mocktime or triggering a `UpdatedBlockTip` refresh) would cross into block `cycle_N + 24` and kick off a DKG phase 1 that the just-started masternodes can't participate in, leading to a null commitment at the next mining window. Shift the target one block earlier to `cycle_N + (DKG_INTERVAL - 2)` = `cycle_N + 22` — the center of the 3-block Idle gap `[cycle_N + 21, cycle_N + 23]`. This leaves a 1-block margin on each side, so harnesses mining up to one extra block stay inside the gap. For the default 8-cycle run this lands at 406 (last orchestrated cycle at 384, tip at `384 + 22 = 406`, `406 % 24 = 22`). Verified via a fresh dashd against the exported datadir: - getblockcount: 406, `406 % 24 = 22` - quorum list: 8 `llmq_test` + 16 `llmq_test_dip0024` rotating quorums, all with non-zero `quorumPublicKey` - No null commits anywhere: `quorum list` only surfaces commits that ProcessCommitment wrote to evoDB, which only happens for non-null commits
The exported directory is now `data/regtest-mn/` instead of `data/regtest-mn-v0.0.1/`. The release tag carries the version, so embedding it in the directory name was redundant and tied the path to a specific version that has to change every release. Matches the convention used by `generate.py` outputs (`regtest-15000/` etc.) — block count or scenario name in the directory, no version tag.
|
Caution Review failedPull request was closed or merged during review 📝 WalkthroughWalkthroughThe PR introduces a complete Dash masternode test data generator for regtest that orchestrates 7 phases: bootstrap controller with wallet and initial blocks, register masternodes via protx, start and sync masternode nodes, enable sporks for DKG/IS/ChainLocks, mine DKG cycles with explicit phase gating, generate SPV test transactions aligned to DKG idle windows, and export network metadata plus node datadirs. ChangesMasternode Test Data Generator
Sequence Diagram(s)sequenceDiagram
participant CLI as CLI Entry<br/>(main)
participant Gen as Phase<br/>Orchestrator
participant MN as MasternodeNetwork<br/>(Controller + MNs)
participant DKG as DKG Cycle<br/>Engine
participant RPC as RPC/Dashd<br/>Instances
CLI->>Gen: Run phases 1-7
Note over Gen: Phase 1: Bootstrap
Gen->>MN: Start controller
Gen->>RPC: Create wallet, mine blocks
Note over Gen: Phase 2: Register MNs
Gen->>RPC: Generate BLS keys
Gen->>RPC: protx register_fund
Note over Gen: Phase 3: Start MNs & Sync
Gen->>MN: Start all masternodes
Gen->>MN: connect_all (peer mesh)
Gen->>RPC: Force mnsync on each MN
Note over Gen: Phase 4: Enable Sporks
Gen->>RPC: Update DKG/IS/ChainLocks sporks
Note over Gen: Phases 5-6: DKG Cycles & Transactions
Gen->>DKG: For each cycle
DKG->>RPC: Mine phase 1 (init)
DKG->>RPC: Wait quorum connections
DKG->>RPC: Mine phases 2-6 (gated)
DKG->>RPC: Verify quorum list
DKG->>RPC: Mine maturity blocks
Gen->>RPC: Generate SPV transactions
Gen->>RPC: Mine to DKG idle window
Note over Gen: Phase 7: Export
Gen->>MN: stop_all, cleanup
Gen->>RPC: Copy regtest datadirs
Gen->>RPC: Export network.json
sequenceDiagram
participant Gen as DKG Cycle<br/>Orchestrator
participant Ctrl as Controller<br/>RPC
participant MN as Masternodes<br/>RPC
participant Chain as Regtest<br/>Chain
Gen->>Ctrl: Align to DKG cycle boundary
Gen->>Ctrl: Mine DIP-0024 warmup (if needed)
Note over Gen: Phase 1: Init
Gen->>Ctrl: Mine block, wait for phase 1 membership
Gen->>Ctrl: Poll received DKG messages
Note over Gen: Quorum Connections
Gen->>Ctrl: Wait for target peer mesh size
Gen->>MN: Check getpeerinfo on each
Note over Gen: Phases 2-6: Contributions & Commitments
loop For phases 2-6
Gen->>Ctrl: Mine block
Gen->>Ctrl: Poll received messages (if gated)
Gen->>Ctrl: Wait for phase advance
end
Note over Gen: Commitment Block
Gen->>Ctrl: Bump mocktime
Gen->>Ctrl: Generate template, mine commitment block
Gen->>Chain: Verify quorum list updated
Note over Gen: Maturity
Gen->>Ctrl: Mine maturity blocks
Gen->>Ctrl: Dump quorum list count
Estimated code review effort🎯 4 (Complex) | ⏱️ ~60 minutes Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Tip 💬 Introducing Slack Agent: The best way for teams to turn conversations into code.Slack Agent is built on CodeRabbit's deep understanding of your code, so your team can collaborate across the entire SDLC without losing context.
Built for teams:
One agent for your entire SDLC. Right inside Slack. Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
…ies (#5) README: - New "Masternode Network" section between "Re-exporting Wallet Data" and "Project Structure", documenting the `regtest-mn.tar.gz` release artifact, the exported directory layout, the `DASHD_MN_DATADIR` env var convention used by `rust-dashcore/dash-spv` integration tests, and `generate_masternode.py` flags. - Project Structure listing now includes `generate_masternode.py` and `generator/masternode_network.py`. CHANGELOG: - New v0.0.4 entry for the masternode network generator (#4) with the asset list (`regtest-mn.tar.gz`, `regtest-40000.tar.gz`, `regtest-200.tar.gz`). - Retroactive v0.0.3 entry for the existing 200-block release (no PR — release-only bump that shipped `regtest-200.tar.gz`).
Summary
Adds
generate_masternode.py, a generator that produces a regtest blockchain with a fully-active 4-masternode network and real DIP-0024 rotating quorums, suitable for SPV masternode-list-sync integration tests. Output ships asdata/regtest-mn/(1 controller + 4 MN datadirs +network.json+ exported wallet stats).What it produces
llmq_test(type 100) andllmq_test_dip0024(type 103), driven through phases 1-6 with message-count gating that mirrors Dash Core'smine_quorum/mine_cycle_quorum.signers,validMembers, andquorumPublicKey.quorum listagainst the exported datadir reports 8 type-100 + 16 type-103 quorums, all real.[cycle_N + 21, cycle_N + 23], past the last cycle's mining window and before the next cycle's phase 1.Notable design points
H + 3Cfor the first rotating quorum to form).addnodemesh seeded before phase 5 so DKG phase-2 contributions don't race the lazy quorum-manager build.DKGCycleErrorwith a per-MN diagnostic dump on timeout (session phase, sent/received message counters,minableCommitmentssnapshot,quorumConnectionscount).Missing rotation ChainLock signatures in QRInfo: sigm0) because Dash Core'sBuildQuorumChainlockInfocan't emit a CL sig for a failed DKG.Verification
Fresh
python3 generate_masternode.py --dashd-path <dashd> --dkg-cycles 8produces an export that, when started in a clean dashd:getblockcount = 406,406 % 24 = 22(Idle gap)quorum listshows 8llmq_test+ 16llmq_test_dip0024quorums, all with non-zeroquorumPublicKeyquorum listonly surfaces commits thatProcessCommitmentwrote to evoDB, and null commits skip that write)Test plan
pre-commitjob passes (lint + unit tests).python3 generate_masternode.py --dashd-path /path/to/dashd --dkg-cycles 8lands tip 406.data/regtest-mn/controllerreportsgetblockcount=406.DASHD_MN_DATADIR=.../regtest-mn.Follow-ups (not in this PR)
Summary by CodeRabbit