Skip to content

test: fix compare benchmarks#358

Merged
maksymar merged 1 commit intomainfrom
maksym/fix-bench-raw
Jun 30, 2025
Merged

test: fix compare benchmarks#358
maksymar merged 1 commit intomainfrom
maksym/fix-bench-raw

Conversation

@maksymar
Copy link
Copy Markdown
Contributor

@maksymar maksymar commented Jun 30, 2025 •
edited
Loading

This PR fixes compare benchmarks, now it correctly measures instructions only inside bench_fn.

Here's the documentation about excluding setup code:
https://docs.rs/canbench-rs/0.2.1/canbench_rs/#excluding-setup-code

@github-actions
Copy link
Copy Markdown

github-actions Bot commented Jun 30, 2025

canbench 🏋 (dir: ./benchmarks/btreemap) ef31b84 2025-06-30 14:21:50 UTC

✅ ./benchmarks/btreemap/canbench_results.yml is up to date
📦 canbench_results_btreemap.csv available in artifacts

---------------------------------------------------

Summary:
  instructions:
    status:   No significant changes 👍
    counts:   [total 285 | regressed 0 | improved 0 | new 0 | unchanged 285]
    change:   [max 0 | p75 0 | median 0 | p25 0 | min 0]
    change %: [max 0.00% | p75 0.00% | median 0.00% | p25 0.00% | min 0.00%]

  heap_increase:
    status:   No significant changes 👍
    counts:   [total 285 | regressed 0 | improved 0 | new 0 | unchanged 285]
    change:   [max 0 | p75 0 | median 0 | p25 0 | min 0]
    change %: [max 0.00% | p75 0.00% | median 0.00% | p25 0.00% | min 0.00%]

  stable_memory_increase:
    status:   No significant changes 👍
    counts:   [total 285 | regressed 0 | improved 0 | new 0 | unchanged 285]
    change:   [max 0 | p75 0 | median 0 | p25 0 | min 0]
    change %: [max 0.00% | p75 0.00% | median 0.00% | p25 0.00% | min 0.00%]

---------------------------------------------------
CSV results saved to canbench_results.csv

@github-actions
Copy link
Copy Markdown

github-actions Bot commented Jun 30, 2025

canbench 🏋 (dir: ./benchmarks/memory_manager) ef31b84 2025-06-30 14:20:08 UTC

✅ ./benchmarks/memory_manager/canbench_results.yml is up to date
📦 canbench_results_memory-manager.csv available in artifacts

---------------------------------------------------

Summary:
  instructions:
    status:   No significant changes 👍
    counts:   [total 3 | regressed 0 | improved 0 | new 0 | unchanged 3]
    change:   [max 0 | p75 0 | median 0 | p25 0 | min 0]
    change %: [max 0.00% | p75 0.00% | median 0.00% | p25 0.00% | min 0.00%]

  heap_increase:
    status:   No significant changes 👍
    counts:   [total 3 | regressed 0 | improved 0 | new 0 | unchanged 3]
    change:   [max 0 | p75 0 | median 0 | p25 0 | min 0]
    change %: [max 0.00% | p75 0.00% | median 0.00% | p25 0.00% | min 0.00%]

  stable_memory_increase:
    status:   No significant changes 👍
    counts:   [total 3 | regressed 0 | improved 0 | new 0 | unchanged 3]
    change:   [max 0 | p75 0 | median 0 | p25 0 | min 0]
    change %: [max 0.00% | p75 0.00% | median 0.00% | p25 0.00% | min 0.00%]

---------------------------------------------------
CSV results saved to canbench_results.csv

@github-actions
Copy link
Copy Markdown

github-actions Bot commented Jun 30, 2025

canbench 🏋 (dir: ./benchmarks/vec) ef31b84 2025-06-30 14:19:55 UTC

✅ ./benchmarks/vec/canbench_results.yml is up to date
📦 canbench_results_vec.csv available in artifacts

---------------------------------------------------

Summary:
  instructions:
    status:   No significant changes 👍
    counts:   [total 16 | regressed 0 | improved 0 | new 0 | unchanged 16]
    change:   [max +20.00K | p75 +20.00K | median 0 | p25 0 | min 0]
    change %: [max +0.62% | p75 +0.55% | median 0.00% | p25 0.00% | min 0.00%]

  heap_increase:
    status:   No significant changes 👍
    counts:   [total 16 | regressed 0 | improved 0 | new 0 | unchanged 16]
    change:   [max 0 | p75 0 | median 0 | p25 0 | min 0]
    change %: [max 0.00% | p75 0.00% | median 0.00% | p25 0.00% | min 0.00%]

  stable_memory_increase:
    status:   No significant changes 👍
    counts:   [total 16 | regressed 0 | improved 0 | new 0 | unchanged 16]
    change:   [max 0 | p75 0 | median 0 | p25 0 | min 0]
    change %: [max 0.00% | p75 0.00% | median 0.00% | p25 0.00% | min 0.00%]

---------------------------------------------------
CSV results saved to canbench_results.csv

@github-actions
Copy link
Copy Markdown

github-actions Bot commented Jun 30, 2025

canbench 🏋 (dir: ./benchmarks/btreeset) ef31b84 2025-06-30 14:20:10 UTC

✅ ./benchmarks/btreeset/canbench_results.yml is up to date
📦 canbench_results_btreeset.csv available in artifacts

---------------------------------------------------

Summary:
  instructions:
    status:   No significant changes 👍
    counts:   [total 100 | regressed 0 | improved 0 | new 0 | unchanged 100]
    change:   [max 0 | p75 0 | median 0 | p25 0 | min 0]
    change %: [max 0.00% | p75 0.00% | median 0.00% | p25 0.00% | min 0.00%]

  heap_increase:
    status:   No significant changes 👍
    counts:   [total 100 | regressed 0 | improved 0 | new 0 | unchanged 100]
    change:   [max 0 | p75 0 | median 0 | p25 0 | min 0]
    change %: [max 0.00% | p75 0.00% | median 0.00% | p25 0.00% | min 0.00%]

  stable_memory_increase:
    status:   No significant changes 👍
    counts:   [total 100 | regressed 0 | improved 0 | new 0 | unchanged 100]
    change:   [max 0 | p75 0 | median 0 | p25 0 | min 0]
    change %: [max 0.00% | p75 0.00% | median 0.00% | p25 0.00% | min 0.00%]

---------------------------------------------------
CSV results saved to canbench_results.csv

@github-actions
Copy link
Copy Markdown

github-actions Bot commented Jun 30, 2025

canbench 🏋 (dir: ./benchmarks/compare) ef31b84 2025-06-30 14:21:00 UTC

✅ ./benchmarks/compare/canbench_results.yml is up to date
📦 canbench_results_compare.csv available in artifacts

---------------------------------------------------

Summary:
  instructions:
    status:   Improvements detected 🟢
    counts:   [total 18 | regressed 0 | improved 17 | new 0 | unchanged 1]
    change:   [max -288.44M | p75 -442.18M | median -708.83M | p25 -1.27B | min -89.21B]
    change %: [max -0.57% | p75 -55.92% | median -72.60% | p25 -82.16% | min -92.32%]

  heap_increase:
    status:   Improvements detected 🟢
    counts:   [total 18 | regressed 0 | improved 18 | new 0 | unchanged 0]
    change:   [max -1.60K | p75 -1.60K | median -1.89K | p25 -3.20K | min -3.78K]
    change %: [max -97.92% | p75 -100.00% | median -100.00% | p25 -100.00% | min -100.00%]

  stable_memory_increase:
    status:   Improvements detected 🟢
    counts:   [total 18 | regressed 0 | improved 15 | new 0 | unchanged 3]
    change:   [max -1 | p75 -129 | median -897 | p25 -1.67K | min -3.20K]
    change %: [max -0.06% | p75 -7.75% | median -53.87% | p25 -100.00% | min -100.00%]

---------------------------------------------------

Only significant changes:
| status | name                     | calls |     ins |  ins Δ% | HI |    HI Δ% |   SMI |   SMI Δ% |
|--------|--------------------------|-------|---------|---------|----|----------|-------|----------|
|   -    | write_chunks_btreemap_1m |       |  88.71B |  -0.57% |  0 | -100.00% | 3.07K |   -4.03% |
|   -    | write_chunks_btreemap_1k |       |   4.21B |  -6.43% |  2 |  -99.88% | 1.54K |   -7.75% |
|   -    | write_chunks_btreemap_1  |       | 362.18M | -44.33% | 34 |  -97.92% | 1.54K |   -7.75% |
|   -    | write_chunks_vec_1m      |       |   1.90B | -48.95% |  0 | -100.00% | 1.54K |   -7.75% |
|   -    | write_chunks_vec_1k      |       | 562.26M | -55.80% |  0 | -100.00% | 1.54K |   -7.75% |
|   -    | write_chunks_vec_1       |       | 549.90M | -56.28% |  0 | -100.00% | 1.54K |   -7.75% |
|   -    | read_chunks_btreemap_1m  |       |  43.95B | -67.00% |  0 | -100.00% |     0 | -100.00% |
|   -    | write_chunks_stable_1    |       | 130.47M | -68.85% |  0 | -100.00% | 1.66K |   -0.06% |
|   -    | write_chunks_stable_1k   |       | 130.60M | -68.91% |  0 | -100.00% | 1.66K |   -0.06% |
|   -    | write_chunks_stable_1m   |       | 255.41M | -76.29% |  0 | -100.00% | 1.66K |   -0.06% |
|   -    | read_chunks_vec_1m       |       |   1.01B | -78.61% |  0 | -100.00% |     0 | -100.00% |
|   -    | read_chunks_stable_1k    |       | 104.99M | -80.04% |  0 | -100.00% |     0 | -100.00% |
|   -    | read_chunks_btreemap_1   |       | 148.72M | -81.41% |  0 | -100.00% |     0 | -100.00% |
|   -    | read_chunks_stable_1m    |       | 230.00M | -82.41% |  0 | -100.00% |     0 | -100.00% |
|   -    | read_chunks_stable_1     |       | 104.86M | -87.10% |  0 | -100.00% |     0 | -100.00% |
|   -    | read_chunks_btreemap_1k  |       | 499.45M | -90.00% |  0 | -100.00% |     0 | -100.00% |
|   -    | read_chunks_vec_1        |       | 104.86M | -92.31% |  0 | -100.00% |     0 | -100.00% |
|   -    | read_chunks_vec_1k       |       | 105.83M | -92.32% |  0 | -100.00% |     0 | -100.00% |

ins = instructions, HI = heap_increase, SMI = stable_memory_increase, Δ% = percent change

---------------------------------------------------
CSV results saved to canbench_results.csv

@maksymar maksymar marked this pull request as ready for review June 30, 2025 14:24
@maksymar maksymar requested a review from a team as a code owner June 30, 2025 14:24
@maksymar maksymar requested a review from Copilot June 30, 2025 14:24
Copilot AI reviewed Jun 30, 2025
View reviewed changes
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR fixes the compare benchmarks by ensuring that measurements correctly capture the instructions executed within bench_fn. Key changes include updating benchmark function signatures to return BenchResult and modifying the bench macros to use #[bench(raw)] for proper result handling.

Reviewed Changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File Description
benchmarks/compare/src/main.rs Updated benchmark functions and macros to return BenchResult.
benchmarks/compare/canbench_results.yml Adjusted benchmark result values to reflect the corrected measurement logic.

@maksymar maksymar merged commit 67dd9fd into main Jun 30, 2025
19 checks passed
@maksymar maksymar deleted the maksym/fix-bench-raw branch June 30, 2025 14:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot Copilot left review comments

@mraszyk mraszyk mraszyk approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@maksymar @mraszyk