Skip to content

ci: split benchmarks per type#316

Merged
maksymar merged 14 commits intomainfrom
maksym/split
May 23, 2025
Merged

ci: split benchmarks per type#316
maksymar merged 14 commits intomainfrom
maksym/split

Conversation

@maksymar
Copy link
Copy Markdown
Contributor

@maksymar maksymar commented May 23, 2025

This PR splits the existing benchmarks into separate per-type benchmarks.

It also updates the CI workflow to run and collect results for each benchmark crate individually.

@maksymar maksymar requested a review from Copilot May 23, 2025 10:17
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR splits the existing unified benchmarks into separate per-crate benchmarks, refactors common code into a shared module, and updates the CI workflow to run and collect results for each benchmark crate individually.

  • Remove the root-level canbench.yml and legacy build scripts
  • Introduce canbench.yml and result files under each of benchmarks/{btreemap,memory_manager,vec}
  • Refactor benchmarks/src/common.rs, add a common module, update imports, and extend CI to handle multiple benchmark jobs

Reviewed Changes

Copilot reviewed 18 out of 18 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
canbench.yml Deleted root-level benchmark config
benchmarks/src/common.rs Made Random trait pub, removed now-obsolete mod lines
benchmarks/vec/src/main.rs Updated use path to benchmarks::common::Random, added stub
benchmarks/vec/canbench.yml Added per-crate benchmark config
benchmarks/vec/canbench_results.yml Committed generated benchmark results
benchmarks/memory_manager/** Added per-crate config, results, and main stub
benchmarks/btreemap/** Added per-crate config, results, and main stub
benchmarks/Cargo.toml Defined lib and three [[bin]] entries for each bench binary
.github/workflows/ci.yml Expanded CI to launch benchmark-btreemap, memory-manager, and vec jobs
Comments suppressed due to low confidence (2)

benchmarks/src/common.rs:1

  • These imports (Blob, FixedVec, Storable) are not used in common.rs and can be removed to reduce unnecessary dependencies.
use ic_stable_structures::storable::{Blob, FixedVec, Storable};

benchmarks/src/common.rs:2

  • The Rand trait is not used in this file; consider removing it and only importing Rng.
use tiny_rng::{Rand, Rng};

Comment thread .github/workflows/ci.yml
Comment thread .github/workflows/ci.yml Outdated
Comment thread .github/workflows/ci.yml Outdated
Comment thread benchmarks/vec/canbench.yml Outdated
@github-actions
Copy link
Copy Markdown

github-actions Bot commented May 23, 2025

canbench 🏋 (dir: ./benchmarks/memory_manager) 58cd834 2025-05-23 11:38:51 UTC

./benchmarks/memory_manager/canbench_results.yml is up to date
📦 canbench_results_memory-manager.csv available in artifacts

---------------------------------------------------

Summary:
  instructions:
    status:   No significant changes detected 👍
    counts:   [total 3 | new 0 | improved 0 | regressed 0 | unchanged 3]
    change:   [min 0 | med 0 | max 0]
    change %: [min 0.00% | med 0.00% | max 0.00%]

  heap_increase:
    status:   No significant changes detected 👍
    counts:   [total 3 | new 0 | improved 0 | regressed 0 | unchanged 3]
    change:   [min 0 | med 0 | max 0]
    change %: [min 0.00% | med 0.00% | max 0.00%]

  stable_memory_increase:
    status:   No significant changes detected 👍
    counts:   [total 3 | new 0 | improved 0 | regressed 0 | unchanged 3]
    change:   [min 0 | med 0 | max 0]
    change %: [min 0.00% | med 0.00% | max 0.00%]

---------------------------------------------------
CSV results saved to canbench_results.csv

@github-actions
Copy link
Copy Markdown

github-actions Bot commented May 23, 2025

canbench 🏋 (dir: ./benchmarks/btreemap) 58cd834 2025-05-23 11:40:29 UTC

./benchmarks/btreemap/canbench_results.yml is up to date
📦 canbench_results_btreemap.csv available in artifacts

---------------------------------------------------

Summary:
  instructions:
    status:   No significant changes detected 👍
    counts:   [total 285 | new 0 | improved 0 | regressed 0 | unchanged 285]
    change:   [min 0 | med 0 | max 0]
    change %: [min 0.00% | med 0.00% | max 0.00%]

  heap_increase:
    status:   No significant changes detected 👍
    counts:   [total 285 | new 0 | improved 0 | regressed 0 | unchanged 285]
    change:   [min 0 | med 0 | max 0]
    change %: [min 0.00% | med 0.00% | max 0.00%]

  stable_memory_increase:
    status:   No significant changes detected 👍
    counts:   [total 285 | new 0 | improved 0 | regressed 0 | unchanged 285]
    change:   [min 0 | med 0 | max 0]
    change %: [min 0.00% | med 0.00% | max 0.00%]

---------------------------------------------------
CSV results saved to canbench_results.csv

@github-actions
Copy link
Copy Markdown

github-actions Bot commented May 23, 2025

canbench 🏋 (dir: ./benchmarks/vec) 58cd834 2025-05-23 11:38:40 UTC

./benchmarks/vec/canbench_results.yml is up to date
📦 canbench_results_vec.csv available in artifacts

---------------------------------------------------

Summary:
  instructions:
    status:   No significant changes detected 👍
    counts:   [total 16 | new 0 | improved 0 | regressed 0 | unchanged 16]
    change:   [min 0 | med 0 | max 0]
    change %: [min 0.00% | med 0.00% | max 0.00%]

  heap_increase:
    status:   No significant changes detected 👍
    counts:   [total 16 | new 0 | improved 0 | regressed 0 | unchanged 16]
    change:   [min 0 | med 0 | max 0]
    change %: [min 0.00% | med 0.00% | max 0.00%]

  stable_memory_increase:
    status:   No significant changes detected 👍
    counts:   [total 16 | new 0 | improved 0 | regressed 0 | unchanged 16]
    change:   [min 0 | med 0 | max 0]
    change %: [min 0.00% | med 0.00% | max 0.00%]

---------------------------------------------------
CSV results saved to canbench_results.csv

@dfinity dfinity deleted a comment from github-actions Bot May 23, 2025
@dfinity dfinity deleted a comment from github-actions Bot May 23, 2025
@dfinity dfinity deleted a comment from github-actions Bot May 23, 2025
@maksymar maksymar changed the title ci: split benchmarks ci: split benchmarks per type May 23, 2025
@maksymar maksymar marked this pull request as ready for review May 23, 2025 11:14
@maksymar maksymar requested a review from a team as a code owner May 23, 2025 11:14
@maksymar maksymar merged commit 938adc4 into main May 23, 2025
13 checks passed
@maksymar maksymar deleted the maksym/split branch May 23, 2025 11:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants