Skip to content

feat: add masternode network generator with real DIP-0024 quorums#4

Merged
xdustinface merged 8 commits into
mainfrom
feat/masternode-generation
May 6, 2026
Merged

feat: add masternode network generator with real DIP-0024 quorums#4
xdustinface merged 8 commits into
mainfrom
feat/masternode-generation

Conversation

@xdustinface
Copy link
Copy Markdown
Contributor

@xdustinface xdustinface commented May 6, 2026

Summary

Adds generate_masternode.py, a generator that produces a regtest blockchain with a fully-active 4-masternode network and real DIP-0024 rotating quorums, suitable for SPV masternode-list-sync integration tests. Output ships as data/regtest-mn/ (1 controller + 4 MN datadirs + network.json + exported wallet stats).

What it produces

  • 8 successful DKG cycles for both llmq_test (type 100) and llmq_test_dip0024 (type 103), driven through phases 1-6 with message-count gating that mirrors Dash Core's mine_quorum / mine_cycle_quorum.
  • Real rotating quorums with non-zero signers, validMembers, and quorumPublicKey. quorum list against the exported datadir reports 8 type-100 + 16 type-103 quorums, all real.
  • Exit tip 406, the center of the DKG Idle gap [cycle_N + 21, cycle_N + 23], past the last cycle's mining window and before the next cycle's phase 1.

Notable design points

  • 3-cycle warmup before the first orchestrated DKG (DIP-0024 needs H + 3C for the first rotating quorum to form).
  • Direct MN<->MN addnode mesh seeded before phase 5 so DKG phase-2 contributions don't race the lazy quorum-manager build.
  • Wait helpers raise DKGCycleError with a per-MN diagnostic dump on timeout (session phase, sent/received message counters, minableCommitments snapshot, quorumConnections count).
  • Masternodes stay running for the entire chain. Null rotating commits would trip rust-dashcore's QRInfo pre-check (Missing rotation ChainLock signatures in QRInfo: sigm0) because Dash Core's BuildQuorumChainlockInfo can't emit a CL sig for a failed DKG.

Verification

Fresh python3 generate_masternode.py --dashd-path <dashd> --dkg-cycles 8 produces an export that, when started in a clean dashd:

  • getblockcount = 406, 406 % 24 = 22 (Idle gap)
  • quorum list shows 8 llmq_test + 16 llmq_test_dip0024 quorums, all with non-zero quorumPublicKey
  • No null commits anywhere in the chain (quorum list only surfaces commits that ProcessCommitment wrote to evoDB, and null commits skip that write)

Test plan

  • CI pre-commit job passes (lint + unit tests).
  • Local end-to-end run: python3 generate_masternode.py --dashd-path /path/to/dashd --dkg-cycles 8 lands tip 406.
  • Tarball extracted, dashd started against data/regtest-mn/controller reports getblockcount=406.
  • Consumer rust-dashcore tests pass with DASHD_MN_DATADIR=.../regtest-mn.

Follow-ups (not in this PR)

  • README section + CHANGELOG entry for v0.0.3.
  • Optional: GH Actions workflow that builds the tarball and uploads on tag push.

Summary by CodeRabbit

  • Chores
    • Added infrastructure for generating and managing Dash masternode test networks on regtest, including multi-node orchestration, synchronization utilities, and network metadata export capabilities.

Multi-node masternode network generator for regtest test data.
Produces 5 datadirs (1 controller + 4 MNs) with DKG cycles and
quorum history for SPV masternode list sync integration testing.

Follows Dash Core test framework ordering:
- mocktime init -> force mnsync -> connect nodes
- setmnthreadactive toggle during peer connections
- DKG phase-by-phase progression with mockscheduler
Remove the -llmqtestinstantsenddip0024=llmq_test_instantsend override
that was switching the DIP0024 InstantSend quorum type from
LLMQ_TEST_DIP0024 (useRotation=true) to LLMQ_TEST_INSTANTSEND
(useRotation=false). This prevented quorum snapshots from being
created during block processing, causing BuildQuorumRotationInfo
to fail with "Cannot find quorum snapshot".

Also use RPC stop for clean node shutdown to flush evoDB.
Rework `phase_5_mine_dkg_cycles` to follow Dash Core's `mine_cycle_quorum` flow from `test/functional/test_framework/test_framework.py`. The previous generator drove only a single `llmq_test` (type 100) session per cycle and never orchestrated the interleaved `llmq_test_dip0024` (type 103) rotating DKG, so every DIP-0024 commitment landed as null and the SPV consumer tests saw zeroed `signers`, `validMembers`, and `quorumPublicKey` for type 103.

Each cycle now:

- advances 3 full DKG intervals before the first run (`cycle_quorum_is_ready` warmup) so rotating quorums can form past `H+3C`, matching `feature_llmq_rotation.py`
- walks phases 1-6 block-by-block, interleaving `q_0` and `q_1` for type 103 while type 100 piggybacks naturally on the shared `dkgInterval=24`
- gates every phase on the expected DKG message counts (`receivedContributions`, `receivedComplaints`, `receivedJustifications`, `receivedPrematureCommitments`) so blocks never advance past a phase before real messages have been exchanged
- mines a single commit block at `cycle+12` via `getblocktemplate + generate(1)`, matching `mine_cycle_quorum`'s terminal step
- verifies all three real quorums via `quorum list` (no `count` arg, so the default returns both `q_0` and `q_1` for rotating types — the prior `count=1` dropped `q_1` from the response even when it was successfully mined)
- mines 8 signing-window maturity blocks

Also seeds direct masternode↔masternode `addnode` connections in `MasternodeNetwork.connect_all` so DKG phase 2 sees real contributions instead of racing against the quorum manager's lazy connection build, and switches node shutdown to RPC `stop` with a SIGTERM fallback so evoDB and quorum snapshots are flushed cleanly.
Consolidate scattered lifecycle helpers and remove dead plumbing uncovered while fixing the DIP-0024 cycle generator.

- Move mocktime control onto `MasternodeNetwork` as `set_mocktime` / `bump_mocktime` / `move_blocks`, plus `all_nodes()`. The previous free functions poked at `network._mocktime` from outside the class and the `hasattr(self, "_mocktime")` guards in `start_masternode_nodes` were paranoid dead checks since `_mocktime` is always set before any node starts.
- Move `force_finish_mnsync` and mocktime application onto `MasternodeNode`.
- Hoist `import subprocess` to the module top (per the project's import-at-top rule).
- Drop the unreachable `find_free_port(controller_p2p_port + ...)` fallback in `start_masternode_nodes`: `allocate_mn_ports()` always runs first, so `mn_p2p_ports` is always populated. Replace with a direct `assert`.
- Replace the `MasternodeConfig` dataclass with direct use of argparse `args`. It was a three-field indirection used exactly once.
- Remove the `completed_cycles` counter and the `dkg_cycles` pass-through from `phase_5_mine_dkg_cycles` to `phase_7_export`. Since `_run_single_dkg_cycle` raises on failure, the counter always equalled `num_cycles`.
- Make `wait_for_quorum_phase`, `wait_for_quorum_connections`, and `wait_for_quorum_list` raise `DKGCycleError` with a diagnostic dump on timeout. This eliminates ~12 call sites of 6-line `_require(...)` boilerplate and pushes the context into the helper that has it.
- Extract `_find_session`, `_find_connection_group`, `_wait_for_dkg_phase`, and `PHASE_GATES` / `PHASE_NAMES` / `LLMQ_TYPE_NUM` tables to collapse duplicated dkgstatus navigation and the per-phase q_0/q_1 interleaving.
- Extract a small `try_addnode` closure in `MasternodeNetwork.connect_all` to replace the four copy-pasted `try/except` addnode blocks.

Net effect: `generate_masternode.py` shrinks from 884 to ~690 lines and `masternode_network.py` from 363 to ~400 lines (extra methods offset by removed boilerplate). Verified end-to-end with `--dkg-cycles 2`: real `llmq_test` and `llmq_test_dip0024` quorums formed in every cycle.
Add `phase_6b_extend_to_quiet_tip` that stops the masternodes and mines
controller-only blocks past the next cycle's mining window (+8 maturity)
before export. For the default 8-cycle run this lands the exported tip at
height 460, matching the formula `current_cycle_start + DKG_INTERVAL +
dkgMiningWindowEnd + SIGN_HEIGHT_OFFSET`.

Previously, phase 6 left the chain tip inside a DKG cycle that had started
but not reached its mining window (tip 412, inside cycle 408's phase 3).
Test harnesses bringing the masternodes back online at that tip would miss
phases 1-3 of the in-progress cycle, fail to contribute, and produce null
commitments when their mine_dkg_cycle extended past the mining window. That
left a future QRInfo diff with no ChainLock sigs for the rotation cycle,
tripping the rust-dashcore QRInfo pre-check "Missing rotation ChainLock
signatures in QRInfo: sigm0" in `test_instantsend_islock_arrives_before_tx`.

Stopping the masternodes before extending guarantees the intermediate
cycles settle as null commitments (no MN to broadcast `qfcommit`, so the
controller's local `minableCommitmentsByQuorum` is empty at mining time
and null commits land deterministically). By the exported tip the mining
window has closed, so null commits are already written to evoDB and the
cycle is no longer "in progress". When the test harness later restarts
the masternodes, it sees a settled chain and can drive a fresh DKG for
the next cycle boundary via `mine_dkg_cycle` without racing a half-lived
DKG.

Extract `DKG_MINING_WINDOW_END` and `SIGN_HEIGHT_OFFSET` as module
constants and use them in phase 5's maturity mine (replacing the inline
`8`) so the window arithmetic lives in one place.
The previous `phase_6b_extend_to_quiet_tip` (added in 4ec740f) shut down masternodes and mined controller-only blocks past the next cycle's mining window to "pre-settle" in-progress DKGs as null commits. That inverts the actual requirement: null rotating commits in the exported chain are precisely what trips rust-dashcore's QRInfo pre-check.

The concrete failure mode, traced against Dash Core and rust-dashcore:

- When the SPV's QRInfo diff covers a rotating cycle whose commitment is null, Dash Core's `BuildQuorumChainlockInfo` walks `new_quorums` and calls `qman.GetQuorum(type, quorumHash)`. Failed DKG commits have no quorum object, so that call returns null and the diff arrives with the null cycle's slot empty in `quorumsCLSigs`.
- rust-dashcore's `apply_diff(mn_list_diff_h)` then cannot pull `sigm0` / `sigm1` / ... for that rotating slot, so `maybe_sigmN = None` and `feed_qr_info` rejects the QRInfo with `Missing rotation ChainLock signatures in QRInfo: sigmN`.

The correct pattern, matching upstream's `mine_quorum` + `move_blocks` convention:

- Keep all masternodes running for the whole chain-gen.
- After the last orchestrated DKG cycle (phase 5 leaves tip at `cycle_N + dkgMiningWindowEnd` = `cycle_N + 20`), send the SPV test transactions and mine just enough blocks to confirm them AND to land the final tip in the DKG Idle gap at `cycle_N + 23`.
- `cycle_N + 23` is the last Idle-phase block before the next cycle's phase 1 at `cycle_N + 24`: no DKG is mid-flight at the exported tip, every rotating commit on-chain is real, and a subsequent QRInfo's work block at `tip - 8 = 399` (for N=384) already sees the latest cycle's commit at `cycle_N + 12 = 396`.

Changes:

- Remove `phase_6b_extend_to_quiet_tip` entirely and drop its call from `main`.
- Rework `phase_6_generate_test_transactions` to send all SPV transactions into the mempool, then mine exactly `DKG_INTERVAL - 1 - DKG_MINING_WINDOW_END` = 3 blocks — lands tip at `cycle_N + 23` and confirms every transaction with ≥1 confirmation. Add an assertion that the entry tip is exactly `cycle_N + DKG_MINING_WINDOW_END` and that the final tip's cycle offset lands in the Idle gap `(DKG_MINING_WINDOW_END, DKG_INTERVAL)`.

Verified end-to-end: tip=407 (407 % 24 = 23 ∈ [21, 23]), 8 real `llmq_test` quorums and 16 real `llmq_test_dip0024` rotating quorums (all with non-zero `quorumPublicKey`), zero null commits anywhere in the chain.
Previously the exit tip landed at `cycle_N + 23` — the last block of the Idle gap, flush against the next cycle's phase 1 at `cycle_N + 24`. Any test harness that mines a single block during its setup before querying the chain (a common pattern for advancing mocktime or triggering a `UpdatedBlockTip` refresh) would cross into block `cycle_N + 24` and kick off a DKG phase 1 that the just-started masternodes can't participate in, leading to a null commitment at the next mining window.

Shift the target one block earlier to `cycle_N + (DKG_INTERVAL - 2)` = `cycle_N + 22` — the center of the 3-block Idle gap `[cycle_N + 21, cycle_N + 23]`. This leaves a 1-block margin on each side, so harnesses mining up to one extra block stay inside the gap. For the default 8-cycle run this lands at 406 (last orchestrated cycle at 384, tip at `384 + 22 = 406`, `406 % 24 = 22`).

Verified via a fresh dashd against the exported datadir:
- getblockcount: 406, `406 % 24 = 22`
- quorum list: 8 `llmq_test` + 16 `llmq_test_dip0024` rotating quorums, all with non-zero `quorumPublicKey`
- No null commits anywhere: `quorum list` only surfaces commits that ProcessCommitment wrote to evoDB, which only happens for non-null commits
The exported directory is now `data/regtest-mn/` instead of `data/regtest-mn-v0.0.1/`. The release tag carries the version, so embedding it in the directory name was redundant and tied the path to a specific version that has to change every release.

Matches the convention used by `generate.py` outputs (`regtest-15000/` etc.) — block count or scenario name in the directory, no version tag.
@coderabbitai
Copy link
Copy Markdown

coderabbitai Bot commented May 6, 2026

Caution

Review failed

Pull request was closed or merged during review

📝 Walkthrough

Walkthrough

The PR introduces a complete Dash masternode test data generator for regtest that orchestrates 7 phases: bootstrap controller with wallet and initial blocks, register masternodes via protx, start and sync masternode nodes, enable sporks for DKG/IS/ChainLocks, mine DKG cycles with explicit phase gating, generate SPV test transactions aligned to DKG idle windows, and export network metadata plus node datadirs.

Changes

Masternode Test Data Generator

Layer / File(s) Summary
Node Management Infrastructure
generator/masternode_network.py (lines 19–168)
MasternodeNode wraps dashd lifecycle: start with regtest flags, poll RPC readiness, set mocktime, force mnsync, and graceful shutdown. find_free_port allocates unused TCP ports for RPC and P2P.
Network Orchestration
generator/masternode_network.py (lines 170–317)
MasternodeNetwork bootstraps controller in temp datadir with SPV/blockfilter flags, then copies regtest state into per-masternode datadirs and starts each with its unique BLS key and shared mocktime.
Peer Mesh & Connectivity
generator/masternode_network.py (lines 317–401)
connect_all disables MN threads, connects controller↔MN bidirectionally and MN↔MN pairwise, re-enables threads, and waits via getpeerinfo polling for target peer count. Lifecycle methods: stop_all, cleanup, generatetoaddress, wait_for_sync.
Constants & DKG Configuration
generate_masternode.py (lines 1–58)
Defines DKG timing (DKG_INTERVAL, DKG_MINING_WINDOW_END), quorum type mappings (type-100 llmq_test, type-103 llmq_test_dip0024), per-phase message-count gates, and executable header.
DKG Diagnostics & Helpers
generate_masternode.py (lines 61–147)
Exception type DKGCycleError and diagnostic helpers: _find_session, _find_connection_group, _dump_dkg_status (dumps per-node DKG sessions, commitments, peer counts), _raise_with_diagnostic (raises errors with status dumps).
Quorum Phase Waiting
generate_masternode.py (lines 150–264)
Polling utilities wait_for_quorum_phase (with optional received-message gating), wait_for_quorum_connections (mesh readiness), and wait_for_quorum_list (verifies quorum presence in quorum list).
Bootstrap & Registration
generate_masternode.py (lines 266–355)
phase_1_bootstrap initializes mocktime, creates/loads wallet, mines blocks for funding. phase_2_register_masternodes allocates MN ports, generates BLS keys/addresses, submits protx register_fund, mines confirmation, and records metadata.
Masternode Startup & Sync
generate_masternode.py (lines 357–440)
phase_3_start_masternodes starts all MNs, reloads controller wallet, reapplies mocktime via RPC, forces mnsync, connects nodes, and mines blocks to enable masternodes. phase_4_enable_sporks updates DKG/IS/ChainLocks sporks and polls propagation.
DKG Cycle Engine
generate_masternode.py (lines 441–550)
_wait_for_dkg_phase drives phase transitions for both quorum types with block mining gated on received messages. _run_single_dkg_cycle aligns to boundaries, mines DIP-0024 warmup, waits phases 1–6 with per-phase gating, mines commitment block using mocktime bump, verifies quorum list presence, and advances with maturity blocks. phase_5_mine_dkg_cycles runs requested cycle count.
Transaction Generation
generate_masternode.py (lines 578–642)
phase_6_generate_test_transactions creates SPV transactions from wallet, mines a configurable block count derived from DKG_INTERVAL and DKG_MINING_WINDOW_END, asserts tip offset is within DKG idle window, and logs generated count.
Export & Cleanup
generate_masternode.py (lines 644–712)
phase_7_export stops nodes, copies controller and MN regtest datadirs to output, exports wallet statistics and network.json metadata (BLS keys, addresses, extra args), and reports total size and chain height/DKG cycle counts.
CLI Entry Point
generate_masternode.py (lines 714–763)
main() parses CLI args (--dashd-path, --dkg-cycles, --output-dir), validates dashd, constructs MasternodeNetwork with spork key and extra args, runs phases 1–7 sequentially with exception handling, and calls network.cleanup() in finally.

Sequence Diagram(s)

sequenceDiagram
    participant CLI as CLI Entry<br/>(main)
    participant Gen as Phase<br/>Orchestrator
    participant MN as MasternodeNetwork<br/>(Controller + MNs)
    participant DKG as DKG Cycle<br/>Engine
    participant RPC as RPC/Dashd<br/>Instances

    CLI->>Gen: Run phases 1-7
    
    Note over Gen: Phase 1: Bootstrap
    Gen->>MN: Start controller
    Gen->>RPC: Create wallet, mine blocks
    
    Note over Gen: Phase 2: Register MNs
    Gen->>RPC: Generate BLS keys
    Gen->>RPC: protx register_fund
    
    Note over Gen: Phase 3: Start MNs & Sync
    Gen->>MN: Start all masternodes
    Gen->>MN: connect_all (peer mesh)
    Gen->>RPC: Force mnsync on each MN
    
    Note over Gen: Phase 4: Enable Sporks
    Gen->>RPC: Update DKG/IS/ChainLocks sporks
    
    Note over Gen: Phases 5-6: DKG Cycles & Transactions
    Gen->>DKG: For each cycle
    DKG->>RPC: Mine phase 1 (init)
    DKG->>RPC: Wait quorum connections
    DKG->>RPC: Mine phases 2-6 (gated)
    DKG->>RPC: Verify quorum list
    DKG->>RPC: Mine maturity blocks
    Gen->>RPC: Generate SPV transactions
    Gen->>RPC: Mine to DKG idle window
    
    Note over Gen: Phase 7: Export
    Gen->>MN: stop_all, cleanup
    Gen->>RPC: Copy regtest datadirs
    Gen->>RPC: Export network.json
Loading
sequenceDiagram
    participant Gen as DKG Cycle<br/>Orchestrator
    participant Ctrl as Controller<br/>RPC
    participant MN as Masternodes<br/>RPC
    participant Chain as Regtest<br/>Chain

    Gen->>Ctrl: Align to DKG cycle boundary
    Gen->>Ctrl: Mine DIP-0024 warmup (if needed)
    
    Note over Gen: Phase 1: Init
    Gen->>Ctrl: Mine block, wait for phase 1 membership
    Gen->>Ctrl: Poll received DKG messages
    
    Note over Gen: Quorum Connections
    Gen->>Ctrl: Wait for target peer mesh size
    Gen->>MN: Check getpeerinfo on each
    
    Note over Gen: Phases 2-6: Contributions & Commitments
    loop For phases 2-6
        Gen->>Ctrl: Mine block
        Gen->>Ctrl: Poll received messages (if gated)
        Gen->>Ctrl: Wait for phase advance
    end
    
    Note over Gen: Commitment Block
    Gen->>Ctrl: Bump mocktime
    Gen->>Ctrl: Generate template, mine commitment block
    Gen->>Chain: Verify quorum list updated
    
    Note over Gen: Maturity
    Gen->>Ctrl: Mine maturity blocks
    Gen->>Ctrl: Dump quorum list count
Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

Poem

🐰 A bustling warren of nodes, all lined up!
Masternodes mine, while the cycles drink up
DKG messages dance through a peer mesh so tight,
Seven phases of order—the test data's just right! 🌙✨

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately describes the main change: adding a masternode network generator with DIP-0024 quorum support, which aligns with the 766+ lines of new functionality in generate_masternode.py and the new MasternodeNetwork/MasternodeNode classes.
Docstring Coverage ✅ Passed Docstring coverage is 86.84% which is sufficient. The required threshold is 80.00%.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/masternode-generation

Tip

💬 Introducing Slack Agent: The best way for teams to turn conversations into code.

Slack Agent is built on CodeRabbit's deep understanding of your code, so your team can collaborate across the entire SDLC without losing context.

  • Generate code and open pull requests
  • Plan features and break down work
  • Investigate incidents and troubleshoot customer tickets together
  • Automate recurring tasks and respond to alerts with triggers
  • Summarize progress and report instantly

Built for teams:

  • Shared memory across your entire org—no repeating context
  • Per-thread sandboxes to safely plan and execute work
  • Governance built-in—scoped access, auditability, and budget controls

One agent for your entire SDLC. Right inside Slack.

👉 Get started


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@xdustinface xdustinface merged commit 7a22e79 into main May 6, 2026
2 of 3 checks passed
@xdustinface xdustinface deleted the feat/masternode-generation branch May 6, 2026 22:23
xdustinface added a commit that referenced this pull request May 6, 2026
…ies (#5)

README:
- New "Masternode Network" section between "Re-exporting Wallet Data" and "Project Structure", documenting the `regtest-mn.tar.gz` release artifact, the exported directory layout, the `DASHD_MN_DATADIR` env var convention used by `rust-dashcore/dash-spv` integration tests, and `generate_masternode.py` flags.
- Project Structure listing now includes `generate_masternode.py` and `generator/masternode_network.py`.

CHANGELOG:
- New v0.0.4 entry for the masternode network generator (#4) with the asset list (`regtest-mn.tar.gz`, `regtest-40000.tar.gz`, `regtest-200.tar.gz`).
- Retroactive v0.0.3 entry for the existing 200-block release (no PR — release-only bump that shipped `regtest-200.tar.gz`).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant