Skip to content

fix: use cargo build locked to remove noise from canbench measurements#327

Merged
maksymar merged 12 commits intomainfrom
maksym/lock
May 28, 2025
Merged

fix: use cargo build locked to remove noise from canbench measurements#327
maksymar merged 12 commits intomainfrom
maksym/lock

Conversation

@maksymar
Copy link
Copy Markdown
Contributor

@maksymar maksymar commented May 28, 2025

This PR uses --locked flag for building benchmarks to avoid 3rd party dependency noise.

It also moves dependency versions into workspace section.

@github-actions
Copy link
Copy Markdown

github-actions Bot commented May 28, 2025

canbench 🏋 (dir: ./benchmarks/memory_manager) 5f6f1c9 2025-05-28 14:29:41 UTC

./benchmarks/memory_manager/canbench_results.yml is up to date
📦 canbench_results_memory-manager.csv available in artifacts

---------------------------------------------------

Summary:
  instructions:
    status:   No significant changes detected 👍
    counts:   [total 3 | new 0 | improved 0 | regressed 0 | unchanged 3]
    change:   [max 0 | p75 0 | median 0 | p25 0 | min 0]
    change %: [max 0.00% | p75 0.00% | median 0.00% | p25 0.00% | min 0.00%]

  heap_increase:
    status:   No significant changes detected 👍
    counts:   [total 3 | new 0 | improved 0 | regressed 0 | unchanged 3]
    change:   [max 0 | p75 0 | median 0 | p25 0 | min 0]
    change %: [max 0.00% | p75 0.00% | median 0.00% | p25 0.00% | min 0.00%]

  stable_memory_increase:
    status:   No significant changes detected 👍
    counts:   [total 3 | new 0 | improved 0 | regressed 0 | unchanged 3]
    change:   [max 0 | p75 0 | median 0 | p25 0 | min 0]
    change %: [max 0.00% | p75 0.00% | median 0.00% | p25 0.00% | min 0.00%]

---------------------------------------------------
CSV results saved to canbench_results.csv

@github-actions
Copy link
Copy Markdown

github-actions Bot commented May 28, 2025

canbench 🏋 (dir: ./benchmarks/btreemap) 5f6f1c9 2025-05-28 14:31:32 UTC

./benchmarks/btreemap/canbench_results.yml is up to date
📦 canbench_results_btreemap.csv available in artifacts

---------------------------------------------------

Summary:
  instructions:
    status:   No significant changes detected 👍
    counts:   [total 285 | new 0 | improved 0 | regressed 0 | unchanged 285]
    change:   [max 0 | p75 0 | median 0 | p25 0 | min 0]
    change %: [max 0.00% | p75 0.00% | median 0.00% | p25 0.00% | min 0.00%]

  heap_increase:
    status:   No significant changes detected 👍
    counts:   [total 285 | new 0 | improved 0 | regressed 0 | unchanged 285]
    change:   [max 0 | p75 0 | median 0 | p25 0 | min 0]
    change %: [max 0.00% | p75 0.00% | median 0.00% | p25 0.00% | min 0.00%]

  stable_memory_increase:
    status:   No significant changes detected 👍
    counts:   [total 285 | new 0 | improved 0 | regressed 0 | unchanged 285]
    change:   [max 0 | p75 0 | median 0 | p25 0 | min 0]
    change %: [max 0.00% | p75 0.00% | median 0.00% | p25 0.00% | min 0.00%]

---------------------------------------------------
CSV results saved to canbench_results.csv

@github-actions
Copy link
Copy Markdown

github-actions Bot commented May 28, 2025

canbench 🏋 (dir: ./benchmarks/compare) 5f6f1c9 2025-05-28 14:30:27 UTC

./benchmarks/compare/canbench_results.yml is up to date
📦 canbench_results_compare.csv available in artifacts

---------------------------------------------------

Summary:
  instructions:
    status:   No significant changes detected 👍
    counts:   [total 18 | new 0 | improved 0 | regressed 0 | unchanged 18]
    change:   [max +65 | p75 0 | median -21 | p25 -56 | min -27.00M]
    change %: [max 0.00% | p75 0.00% | median -0.00% | p25 -0.00% | min -0.72%]

  heap_increase:
    status:   No significant changes detected 👍
    counts:   [total 18 | new 0 | improved 0 | regressed 0 | unchanged 18]
    change:   [max 0 | p75 0 | median 0 | p25 0 | min 0]
    change %: [max 0.00% | p75 0.00% | median 0.00% | p25 0.00% | min 0.00%]

  stable_memory_increase:
    status:   No significant changes detected 👍
    counts:   [total 18 | new 0 | improved 0 | regressed 0 | unchanged 18]
    change:   [max 0 | p75 0 | median 0 | p25 0 | min 0]
    change %: [max 0.00% | p75 0.00% | median 0.00% | p25 0.00% | min 0.00%]

---------------------------------------------------
CSV results saved to canbench_results.csv

@github-actions
Copy link
Copy Markdown

github-actions Bot commented May 28, 2025

canbench 🏋 (dir: ./benchmarks/vec) 5f6f1c9 2025-05-28 14:29:29 UTC

./benchmarks/vec/canbench_results.yml is up to date
📦 canbench_results_vec.csv available in artifacts

---------------------------------------------------

Summary:
  instructions:
    status:   No significant changes detected 👍
    counts:   [total 16 | new 0 | improved 0 | regressed 0 | unchanged 16]
    change:   [max 0 | p75 0 | median 0 | p25 0 | min 0]
    change %: [max 0.00% | p75 0.00% | median 0.00% | p25 0.00% | min 0.00%]

  heap_increase:
    status:   No significant changes detected 👍
    counts:   [total 16 | new 0 | improved 0 | regressed 0 | unchanged 16]
    change:   [max 0 | p75 0 | median 0 | p25 0 | min 0]
    change %: [max 0.00% | p75 0.00% | median 0.00% | p25 0.00% | min 0.00%]

  stable_memory_increase:
    status:   No significant changes detected 👍
    counts:   [total 16 | new 0 | improved 0 | regressed 0 | unchanged 16]
    change:   [max 0 | p75 0 | median 0 | p25 0 | min 0]
    change %: [max 0.00% | p75 0.00% | median 0.00% | p25 0.00% | min 0.00%]

---------------------------------------------------
CSV results saved to canbench_results.csv

@maksymar maksymar marked this pull request as ready for review May 28, 2025 13:18
@maksymar maksymar requested a review from a team as a code owner May 28, 2025 13:18
@maksymar maksymar requested a review from Copilot May 28, 2025 13:18
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR standardizes dependency versions across the workspace and ensures benchmark builds use the lockfile for consistent measurements.

  • Added --locked to all CanBench build_cmd entries to remove noise from benchmark outputs
  • Migrated many crate dependencies to workspace-managed versions and bumped key versions (e.g., serde, ic-cdk, candid)
  • Reorganized per-example Cargo.toml files to use *.workspace = true where appropriate

Reviewed Changes

Copilot reviewed 15 out of 15 changed files in this pull request and generated no comments.

Show a summary per file
File Description
fuzz/Cargo.toml Bumped serde version and reordered dependencies
examples/src/vecs_and_strings/Cargo.toml Enabled ic-cdk as a workspace dependency
examples/src/task_timer/Cargo.toml Switched ic0 to workspace dependency and reordered entries
examples/src/quick_start/Cargo.toml Converted ciborium and ic-cdk to workspace dependencies
examples/Cargo.toml Updated workspace dependency versions (candid, ic-cdk, etc.)
benchmarks/vec/canbench.yml Added --locked flag to build_cmd
benchmarks/memory_manager/canbench.yml Added --locked flag to build_cmd
benchmarks/compare/canbench_results.yml Updated instruction counts after re-running with --locked
benchmarks/compare/canbench.yml Added --locked flag to build_cmd
benchmarks/btreemap/canbench.yml Added --locked flag to build_cmd
benchmarks/Cargo.toml Migrated maplit, serde, tiny-rng, and ic-cdk to workspace
Cargo.toml Switched many deps (e.g., ic_principal, hex, proptest) to workspace and updated workspace versions

@maksymar maksymar marked this pull request as draft May 28, 2025 13:34
@maksymar maksymar marked this pull request as ready for review May 28, 2025 13:34
@maksymar maksymar marked this pull request as draft May 28, 2025 13:36
@maksymar maksymar marked this pull request as ready for review May 28, 2025 13:37
@maksymar maksymar merged commit 75f1c8f into main May 28, 2025
14 checks passed
@maksymar maksymar deleted the maksym/lock branch May 28, 2025 17:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants