A simple web service that records performance numbers for a software project, split into branches and commits. Intended to be used in tandem with CI.
The pages are listed below:
| Feature | Description | Screenshot |
|---|---|---|
| Branch list | Populated as CI sends results to it. |  |
| Branch charts | On a given branch, it will chart the results over time. |  |
| Commit list | You can see the last 28 commits (configurable via ?limit=.. query param). |
 |
| Compare commits | Diff individual commits. |  |
Home page is the branch list.
Data is stored in a simple SQLite database.
The URL to send data is: https://your-deployed-perfly/branch/$BRANCH_NAME/$COMMIT_HASH?token=
The project also provides a CLI tool that renders benchmark graphs to a standalone HTML file (using the same Plotly graph style as the web UI) and opens it locally.
cabal build benchmark-display
Install benchmark-display to a user's bin directory:
mkdir -p "$HOME/.local/bin"
cabal install benchmark-display \
--install-method=copy \
--installdir="$HOME/.local/bin" \
--overwrite-policy=alwaysMake sure your shell PATH includes that directory (for zsh):
echo 'export PATH="$HOME/.local/bin:$PATH"' >> "$HOME/.zshrc"
source "$HOME/.zshrc"Then you can run:
benchmark-display --helpIf you prefer not to install, use cabal run benchmark-display -- ...
from the project directory.
Run with JSON files (each file must be a top-level JSON array of
Benchmark values, not a Commit object):
benchmark-display run-1.json run-2.jsonWrite to a custom output path:
benchmark-display --output reports/benchmark-display.html run-1.json run-2.jsonRead data from SQLite (same DB model as the web server):
benchmark-display --sqlite perf.sqlite3 --branch master --limit 28If you did not install to the path, the commands need to be modified like this:
cabal run benchmark-display -- --output benchmark.html run-1.json run-2.jsonNotes:
- The generated file defaults to
benchmark-display.html. - After writing, the tool runs
open benchmark-display.htmlon macOS (orxdg-openon Linux). - X-axis labels are derived from input files in CLI argument order.
- If a filename ends with
-<hex>.json, that<hex>suffix is used as the label; otherwise the.json-stripped basename is used.
Examples of labels:
bench-master-1e2a4b.json->1e2a4bbenchmark_snapshot.json->benchmark_snapshot
The simple idea is that for a given commit we do some benchmarks.
- For each benchmark there is a subject (the thing being tested) and a list of tests about that subject.
- A test is a tuple of a list of factors and a set of metrics collected.
- You might test a subject with a few different combinations of factors, which would yield a different set of metrics.
- A factor in this schema is literally just a key/value pair of text fields; you can put whatever you want. But it might be "iterations" or "number of users" or "size of blah".
- A metric consists of a name of a thing being measured (time, space, faults, whatever) and then the following numbers: an upper/lower range, mean and standard deviation.
See below for a JSON schema. There is an example in the next section.
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"title": "Commit",
"type": "object",
"properties": {
"branch": {
"type": "string",
"description": "What branch this was on"
},
"commit": {
"type": "string",
"description": "Commit"
},
"result": {
"$ref": "#/$defs/Result"
}
},
"required": ["branch", "commit", "result"],
"$defs": {
"Result": {
"type": "object",
"properties": {
"benchmarks": {
"type": "array",
"items": { "$ref": "#/$defs/Benchmark" }
}
},
"required": ["benchmarks"]
},
"Benchmark": {
"type": "object",
"properties": {
"subject": {
"type": "string",
"description": "What is being tested"
},
"tests": {
"type": "array",
"items": { "$ref": "#/$defs/Test" }
}
},
"required": ["subject", "tests"]
},
"Test": {
"type": "object",
"properties": {
"factors": {
"type": "array",
"items": { "$ref": "#/$defs/Factor" }
},
"metrics": {
"type": "array",
"items": { "$ref": "#/$defs/Metric" }
}
},
"required": ["factors", "metrics"]
},
"Factor": {
"type": "object",
"properties": {
"factor": {
"type": "string",
"description": "What aspect is being varied"
},
"value": {
"type": "string",
"description": "The value of this factor (can represent both text and numbers)"
}
},
"required": ["factor", "value"]
},
"Metric": {
"type": "object",
"properties": {
"metric": {
"type": "string",
"description": "Name of the metric (e.g., 'time_ms')"
},
"rangeLower": {
"type": "number",
"description": "Lower bound of the range"
},
"rangeUpper": {
"type": "number",
"description": "Upper bound of the range"
},
"mean": {
"type": "number",
"description": "Mean value"
},
"stddev": {
"type": "number",
"description": "Standard deviation"
}
},
"required": ["metric", "rangeLower", "rangeUpper", "mean", "stddev"]
}
}
}Below is an example of two test subjects with different factors each.
Expand for full JSON example.
{
"branch": "master",
"commit": "997d6b8c3cf3c5ea14c40ec8c0f55f9c574a51cc",
"result": {
"benchmarks": [
{
"subject": "create things",
"tests": [
{
"factors": [
{
"factor": "records",
"value": "2"
}
],
"metrics": [
{
"mean": 0.1,
"metric": "time",
"rangeLower": 0.1,
"rangeUpper": 1.1,
"stddev": 0.1
},
{
"mean": 610.1,
"metric": "allocated MiB",
"rangeLower": 260.1,
"rangeUpper": 1660.1,
"stddev": 699.1
},
{
"mean": 433.1,
"metric": "peak allocated MiB",
"rangeLower": 433.1,
"rangeUpper": 433.1,
"stddev": 0.1
}
]
},
{
"factors": [
{
"factor": "records",
"value": "20"
}
],
"metrics": [
{
"mean": 4.1,
"metric": "time",
"rangeLower": 4.1,
"rangeUpper": 4.1,
"stddev": 0.1
},
{
"mean": 2605.1,
"metric": "allocated MiB",
"rangeLower": 2604.1,
"rangeUpper": 2606.1,
"stddev": 0.1
},
{
"mean": 471.1,
"metric": "peak allocated MiB",
"rangeLower": 444.1,
"rangeUpper": 489.1,
"stddev": 19.1
}
]
},
{
"factors": [
{
"factor": "records",
"value": "200"
}
],
"metrics": [
{
"mean": 41.1,
"metric": "time",
"rangeLower": 41.1,
"rangeUpper": 41.1,
"stddev": 0.1
},
{
"mean": 26074.1,
"metric": "allocated MiB",
"rangeLower": 26074.1,
"rangeUpper": 26074.1,
"stddev": 0.1
},
{
"mean": 499.1,
"metric": "peak allocated MiB",
"rangeLower": 493.1,
"rangeUpper": 505.1,
"stddev": 5.1
}
]
}
]
},
{
"subject": "Do thing",
"tests": [
{
"factors": [
{
"factor": "iterations",
"value": "20"
},
{
"factor": "datapoints",
"value": "1"
}
],
"metrics": [
{
"mean": 2.1,
"metric": "time",
"rangeLower": 2.1,
"rangeUpper": 2.1,
"stddev": 0.1
},
{
"mean": 2037.1,
"metric": "allocated MiB",
"rangeLower": 2036.1,
"rangeUpper": 2038.1,
"stddev": 0.1
},
{
"mean": 1674.1,
"metric": "peak allocated MiB",
"rangeLower": 688.1,
"rangeUpper": 2700.1,
"stddev": 633.1
}
]
}
]
}
]
}
}An example request:
POST http://localhost:8080/hooks/update?token=apitok
Content-Type: application/json
{
"branch": "master",
"commit": "ccc0f55f9c574a518997d6b8c3cf3c5ea14c40ec",
"result": {
"benchmarks": [
...
Example systemd configuration. The two environment variables to
configure are PERF_TOKEN and PORT. It needs write access to the
PWD to write a perf.sqlite3 database file.
[Unit]
Description=perfly
After=network.target
[Service]
Type=simple
WorkingDirectory=/home/you/perfly-repo/
ExecStart=/usr/bin/env perfly
Restart=on-failure
Environment="PATH=/home/you/.cabal/bin"
Environment="PERF_TOKEN=<your token>"
Environment="PORT=8080"
[Install]
WantedBy=multi-user.target
You can deploy with HTTPS and OAuth2 protection using Caddy or similar.