Skip to content

Disk-spill for proving 32M program#466

Merged
MauroToscano merged 254 commits into
mainfrom
feat/disk-spill-v2-parallel-r2
May 19, 2026
Merged

Disk-spill for proving 32M program#466
MauroToscano merged 254 commits into
mainfrom
feat/disk-spill-v2-parallel-r2

Conversation

@gabrielbosio
Copy link
Copy Markdown
Collaborator

@gabrielbosio gabrielbosio commented Mar 25, 2026

Enables proving programs that exceed available RAM by spilling intermediate data (traces, LDE evaluations, Merkle trees) to disk via mmap.

Extends #444 with reduced time regression.

On a 128 GB machine with 4 GB of swap, the VM can prove fib_iterative_16M on main and cannot prove fib_iterative_32M because of OOM error. This PR allows the VM to prove fib_iterative_32M.

Gated behind disk-spill cargo feature.

Enables logging by making existing log::* function calls to actually log the expected messages and adds logging to indicate storage mode used when disk spill is enabled.

Bench Results

Program: fib_iterative_32M.elf
Command executed 3 times:

./target/release/cli prove executor/program_artifacts/asm/fib_iterative_32M.elf -o /tmp/proof.bin --time

Proving time: ~173s

@gabrielbosio
Copy link
Copy Markdown
Collaborator Author

/bench

Comment thread crypto/stark/src/prover.rs Outdated
@github-actions
Copy link
Copy Markdown

github-actions Bot commented Mar 25, 2026

Benchmark — fib_iterative_8M (median of 3)

Table parallelism: auto (cores / 3)

Metric main PR Δ
Peak heap 53104 MB 52611 MB -493 MB (-0.9%) ⚪
Prove time 25.197s 25.376s +0.179s (+0.7%) ⚪

✅ No significant change.

✅ Low variance (time: 0.9%, heap: 1.0%)

Commit: c5a41dc · Baseline: built from main · Runner: self-hosted bench

@gabrielbosio
Copy link
Copy Markdown
Collaborator Author

/bench

@gabrielbosio
Copy link
Copy Markdown
Collaborator Author

/bench

@gabrielbosio
Copy link
Copy Markdown
Collaborator Author

/bench

@gabrielbosio
Copy link
Copy Markdown
Collaborator Author

/bench

@gabrielbosio gabrielbosio changed the title Parallelize disk-spill Rounds 2-4 proving loop Simplify disk-spill to trace-only spilling Mar 26, 2026
@gabrielbosio gabrielbosio marked this pull request as ready for review March 26, 2026 20:52
@gabrielbosio
Copy link
Copy Markdown
Collaborator Author

/bench

@gabrielbosio
Copy link
Copy Markdown
Collaborator Author

/bench

@gabrielbosio gabrielbosio force-pushed the feat/disk-spill-v2-parallel-r2 branch from 0a9091c to 0f35e6d Compare March 31, 2026 14:56
@gabrielbosio
Copy link
Copy Markdown
Collaborator Author

/codex /claude

Comment thread crypto/crypto/src/mmap_util.rs
Comment thread crypto/crypto/src/mmap_util.rs
@yetanotherco yetanotherco deleted a comment from claude Bot May 7, 2026
@yetanotherco yetanotherco deleted a comment from github-actions Bot May 7, 2026
@gabrielbosio gabrielbosio changed the title Disk-spill for proving 24M program Disk-spill for proving 32M program May 7, 2026
@gabrielbosio gabrielbosio marked this pull request as ready for review May 7, 2026 20:35
@yetanotherco yetanotherco deleted a comment from github-actions Bot May 7, 2026
@gabrielbosio gabrielbosio force-pushed the feat/disk-spill-v2-parallel-r2 branch from 18322f5 to fafc9a5 Compare May 19, 2026 15:23
@MauroToscano MauroToscano added this pull request to the merge queue May 19, 2026
Merged via the queue into main with commit 46bc9be May 19, 2026
12 checks passed
@MauroToscano MauroToscano deleted the feat/disk-spill-v2-parallel-r2 branch May 19, 2026 18:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants