Skip to content

feat: add statistical convergence and outlier detection#26

Merged
nazarhussain merged 13 commits intomainfrom
nh/convergence-update
Jan 28, 2025
Merged

feat: add statistical convergence and outlier detection#26
nazarhussain merged 13 commits intomainfrom
nh/convergence-update

Conversation

@nazarhussain
Copy link
Copy Markdown

Motivation

Make sure the benchmark have stable results in the unstable environments.

Description

  • Add statistical convergence algorithm to stop the benchmark earliest possible
  • Add an outlier cleanup option before calculating the average

Steps to test or reproduce

  • Run all tests

@github-actions
Copy link
Copy Markdown

github-actions bot commented Jan 27, 2025

Performance Report

✔️ no performance regression detected

Full benchmark results
Benchmark suite Current: 7e27781 Previous: null Ratio
sum array with raw for loop 933.71 us/op
sum array with reduce 10.157 ms/op
sum array with reduce beforeEach 82.641 us/op
sum array with reduce before beforeEach 81.987 us/op
sum array with reduce high threshold 10.241 ms/op
sum array with reduce no threshold 10.209 ms/op

by benchmarkbot/action

/** Trigger GC cleanup every test to have consistent memory usage */
triggerGC?: boolean;
/** The algorithm to detect the convergence to stop the benchmark function runs */
convergence?: "linear" | "cv";
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what does cv mean? maybe include a brief description of each in the docstring

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added the detailed comments.

@nazarhussain nazarhussain merged commit 7efbd6f into main Jan 28, 2025
@nazarhussain nazarhussain deleted the nh/convergence-update branch January 28, 2025 16:48
This was referenced Apr 30, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants