Skip to content

feat: add parallel compilation with Rayon#10

Merged
ryanorendorff merged 1 commit intomainfrom
parallel-compile
Oct 16, 2025
Merged

feat: add parallel compilation with Rayon#10
ryanorendorff merged 1 commit intomainfrom
parallel-compile

Conversation

@ryanorendorff
Copy link
Contributor

Refactor preprocessor to compile code blocks in parallel using Rayon thread pool, achieving 2.46x speedup (1.24s -> 0.50s for 13 blocks in the test fixtures).

Module refactoring:

  • Add compilation.rs: CompilationTask/Result types and parallel execution
  • Add task_collector.rs: Task collection with validation limits
  • Add reporting.rs: Error formatting and compilation statistics

Key improvements:

  • Add parallel_jobs config option to control thread count
  • Add CompilationTask::compile method for zero-copy field moves
  • Use Arc for chapter paths to avoid cloning per task
  • Collect all errors before failing to show all issues at once
  • Add compilation statistics with per-language timing breakdown

Performance:

  • Statistics show total time, average per block, and per-language breakdown
  • Individual block timings available at debug log level

Refactor preprocessor to compile code blocks in parallel using Rayon
thread pool, achieving 2.46x speedup (1.24s -> 0.50s for 13 blocks).

Module refactoring:
- Split monolithic preprocessor.rs (487 lines) into focused modules
- Add compilation.rs: CompilationTask/Result types and parallel execution
- Add task_collector.rs: Task collection with validation limits
- Add reporting.rs: Error formatting and compilation statistics

Key improvements:
- Add parallel_jobs config option to control thread count
- Add CompilationTask::compile method for zero-copy field moves
- Use Arc<PathBuf> for chapter paths to avoid cloning per task
- Collect all errors before failing to show all issues at once
- Add compilation statistics with per-language timing breakdown

Performance:
- Statistics show total time, average per block, and per-language breakdown
- Individual block timings available at debug log level
@ryanorendorff ryanorendorff merged commit 5e3b3b1 into main Oct 16, 2025
5 checks passed
@ryanorendorff ryanorendorff deleted the parallel-compile branch October 16, 2025 07:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant