Skip to content

Add benchmark for rispy#65

Merged
J535D165 merged 7 commits intoMrTango:mainfrom
J535D165:add-rispy-benchmark
May 20, 2025
Merged

Add benchmark for rispy#65
J535D165 merged 7 commits intoMrTango:mainfrom
J535D165:add-rispy-benchmark

Conversation

@J535D165
Copy link
Copy Markdown
Collaborator

This PR introduces a simple benchmark tool for the performance of rispy. The benchmark consists of 10k identical records, which are parsed 100 times.

I will use this PR in an upcoming PR with performance improvements.

@holub008 @shapiromatron @PeterLombaers

@PeterLombaers
Copy link
Copy Markdown

It might be a good idea to use a tool like pytest-benchmark when benchmarking pieces of code. That makes the benchmark results a bit more reliable, it will warm up your machine before running the benchmark, run it multiple times and calculate error margins based on that.

@J535D165
Copy link
Copy Markdown
Collaborator Author

Nice tool, I wanted to avoid introducing new dependencies, but this one looks worth it. I am happy to implement it now or later. Can the maintainers @shapiromatron @MrTango point us in the right direction?

Btw, I published some results in PR #66.

@shapiromatron
Copy link
Copy Markdown
Collaborator

Let's implement now in this PR; that seems like a worthwhile dev dependency. Looking forward to this and the other PRs that you're queuing up!

@J535D165
Copy link
Copy Markdown
Collaborator Author

Thanks for the suggestion @PeterLombaers. Can you review @shapiromatron?

I disabled the single tests for now as they mess up the comparison table a bit (us vs ms).

--------------------------------------------------------------------------------------------- benchmark: 2 tests --------------------------------------------------------------------------------------------
Name (time in ms)                             Min                 Max                Mean             StdDev              Median                IQR            Outliers     OPS            Rounds  Iterations
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_benchmark_rispy_large               484.5452 (1.0)      517.4370 (1.0)      502.2159 (1.0)      14.4051 (1.0)      499.9848 (1.0)      25.6566 (1.0)           2;0  1.9912 (1.0)           5           1
test_benchmark_rispy_large_multiline     569.1948 (1.17)     626.2228 (1.21)     592.6684 (1.18)     27.2004 (1.89)     581.3098 (1.16)     50.2680 (1.96)          1;0  1.6873 (0.85)          5           1
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Legend:
  Outliers: 1 Standard Deviation from Mean; 1.5 IQR (InterQuartile Range) from 1st Quartile and 3rd Quartile.
  OPS: Operations Per Second, computed as 1 / Mean

@shapiromatron shapiromatron self-assigned this Mar 26, 2025
Copy link
Copy Markdown
Collaborator

@shapiromatron shapiromatron left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM - I had two minor comments to address, but it looks fine, after those two comments are addressed.

Comment thread tests/test_benchmark.py Outdated
Comment thread Makefile Outdated
@J535D165
Copy link
Copy Markdown
Collaborator Author

Good, will fix this over the weekend.

@shapiromatron
Copy link
Copy Markdown
Collaborator

It feels ironic after you waited 6+ months for me to review this, but now that there's a little momentum, could you fix please when you have a moment? I'd like to cut a new release soon. Thank you!

@J535D165 J535D165 merged commit 32280d0 into MrTango:main May 20, 2025
5 checks passed
@J535D165 J535D165 deleted the add-rispy-benchmark branch May 20, 2025 17:29
@J535D165
Copy link
Copy Markdown
Collaborator Author

Done :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants