Skip to content

bench: add A/B branch comparison workflow to BENCHMARKING.md#92721

Merged
timneutkens merged 2 commits into
canaryfrom
04-13-bench_add_ab_branch_comparison_workflow
Apr 13, 2026
Merged

bench: add A/B branch comparison workflow to BENCHMARKING.md#92721
timneutkens merged 2 commits into
canaryfrom
04-13-bench_add_ab_branch_comparison_workflow

Conversation

@timneutkens
Copy link
Copy Markdown
Contributor

@timneutkens timneutkens commented Apr 13, 2026

What

Adds a new "A/B branch comparison" section to bench/BENCHMARKING.md with practical guidance for comparing benchmark results across two branches.

Why

From running benchmarks comparing canary vs PR #92678, we identified several patterns that lead to noisy or misleading results:

  • Running the full route suite when only one route shows signal wastes time (~3 min per run)
  • The default 120 serial requests is too noisy for sub-2ms routes
  • Single runs can swing 10-15% on light routes
  • Percentage deltas from a single pair of runs can be misleading without absolute numbers

What's in the new section

  • Start with a focused route, not the full suite
  • Increase request counts for fast routes (500 serial / 5000 load)
  • Run at least 3 times per side to average out noise
  • Compare absolute req/s, not just deltas
  • Watch for system state drift between runs
  • Full example workflow with checkout/build/run loop

@nextjs-bot nextjs-bot added created-by: Turbopack team PRs by the Turbopack team. tests labels Apr 13, 2026
Copy link
Copy Markdown
Contributor Author

This stack of pull requests is managed by Graphite. Learn more about stacking.

@timneutkens timneutkens marked this pull request as ready for review April 13, 2026 11:38
@timneutkens timneutkens merged commit 49dd903 into canary Apr 13, 2026
77 checks passed
@timneutkens timneutkens deleted the 04-13-bench_add_ab_branch_comparison_workflow branch April 13, 2026 11:56
@github-actions github-actions Bot locked as resolved and limited conversation to collaborators Apr 27, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants