ARROW-5071: [Archery] Implement running benchmark suite #4249
Conversation
The C++ benchmark would not provide feedback due to capture stdout JSON result. Refactored this such that benchmarks outputs to both stdout in text format and json in a named filed.
Supports writing the result into a file. This will allow more stable composition.
Allows running benchmark suites for a specific build.
|
@pitrou this adds your requested feature to compare against offline data. |
- Implement json serialization for Benchmark and BenchmarkSuite
fsaintjacques
force-pushed
the
ARROW-5071-benchmark-run
branch
from
29034c6 to
1ca0aa9
Compare
Codecov Report
@@ Coverage Diff @@
## master #4249 +/- ##
===========================================
- Coverage 88.19% 63.62% -24.58%
===========================================
Files 660 375 -285
Lines 72992 53019 -19973
Branches 1251 0 -1251
===========================================
- Hits 64378 33735 -30643
- Misses 8501 19284 +10783
+ Partials 113 0 -113
Continue to review full report at Codecov.
|
|
Should the JSON benchmark output format be documented? Any special instructions for those writing or modifying benchmarks (naming, how to maintain continuity with historical data and when to break it, etc.) so that they work correctly in this system? |
|
I don't think it's worth documenting yet, as the goal is to simply pass the serialized state from one archery sub-command to another. Adding documentation on writing/modifying benchmarks is definitively worth it. It reminds me that we need to add version to benchmark metadata once we want to track this in a (persisted) database. |
|
Ok. How necessary is version in the benchmark metadata itself? Isn't version knowable based on timestamp? |
|
I'd say git commit is a better proxy for version. The problem with having only the proxy (say commit or timestamp), is that you can't decided whether 2 samples are comparable. By version, I didn't mean an actual human/semver version, but a key to differentiate if 2 named benchmarks are comparable. Usually, the shasum of the function body (file in our case because it's the easiest proxy) will dictate if the benchmarks are comparable. Note that this is low-medium priority as benchmark body don't change often, and you can always work around this by renaming the benchmark (making the old version not comparable with the new name) and making this an official policy. |
4 participants
archery benchmark runsub-command that performs a single run without comparing to another run. This is the main basic blocks to run a benchmark and publish the result to a database, e.g. codespeed and/or custom db. Push is not provided in this PR.from_rev_or_pathmethod to also support the previous point json output. Thus, you can compare a mixture of commit(s)/build(s)/run's output(s). Effectively adding support for comparing on cached results of a run.--output,-means stdout (the defaults).GoogleBenchmarkRunnerto output partial results to stdout for progress feedback.