From 38f2d14cddf980a576027345e126b41e14e69e11 Mon Sep 17 00:00:00 2001 From: Phanindra899 Date: Mon, 30 Mar 2026 17:38:16 +0530 Subject: [PATCH 1/6] Add structured documentation homepage replacing directory listing --- docs/index.html | 107 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 107 insertions(+) create mode 100644 docs/index.html diff --git a/docs/index.html b/docs/index.html new file mode 100644 index 0000000..a3a4199 --- /dev/null +++ b/docs/index.html @@ -0,0 +1,107 @@ + + + + + Apache Otava Documentation + + + + + +
+ +

Apache Otava Documentation

+ +

+ Structured entry point to the Otava documentation, + replacing the default directory listing. +

+ +
+

Documentation

+ +
+ +
+

Project

+ +
+ +
+ + From d876ce6a526a9a04caab91584a02b666ab5143e3 Mon Sep 17 00:00:00 2001 From: Phanindra899 Date: Mon, 30 Mar 2026 17:47:50 +0530 Subject: [PATCH 2/6] Add documentation landing page replacing directory listing --- docs/index.html | 187 +++++++++++++++++++++++------------------------- 1 file changed, 90 insertions(+), 97 deletions(-) diff --git a/docs/index.html b/docs/index.html index a3a4199..7c8201e 100644 --- a/docs/index.html +++ b/docs/index.html @@ -1,107 +1,100 @@ - - - Apache Otava Documentation - + + Apache Otava Documentation + + -
- -

Apache Otava Documentation

- -

- Structured entry point to the Otava documentation, - replacing the default directory listing. -

+
+ +

Apache Otava Documentation

+ +

+ Structured entry point to the Otava documentation, replacing the default directory listing. +

+ +
+

Documentation

+ +
+ +
+

Project

+ +
-
-

Documentation

-
- -
-

Project

- -
- -
- + + \ No newline at end of file From 67fc0a97dfbff4f8f397fc0fe6278b2755e81b42 Mon Sep 17 00:00:00 2001 From: Phanindra899 Date: Tue, 31 Mar 2026 03:54:43 +0530 Subject: [PATCH 3/6] Add initial Sphinx documentation setup with MyST support --- docs_sphinx/Makefile | 20 +++ docs_sphinx/make.bat | 35 +++++ docs_sphinx/source/basics.md | 197 ++++++++++++++++++++++++++ docs_sphinx/source/big_query.md | 43 ++++++ docs_sphinx/source/conf.py | 32 +++++ docs_sphinx/source/csv.md | 64 +++++++++ docs_sphinx/source/getting_started.md | 154 ++++++++++++++++++++ docs_sphinx/source/grafana.md | 84 +++++++++++ docs_sphinx/source/graphite.md | 125 ++++++++++++++++ docs_sphinx/source/index.rst | 17 +++ docs_sphinx/source/install.md | 36 +++++ docs_sphinx/source/postgresql.md | 138 ++++++++++++++++++ 12 files changed, 945 insertions(+) create mode 100644 docs_sphinx/Makefile create mode 100644 docs_sphinx/make.bat create mode 100644 docs_sphinx/source/basics.md create mode 100644 docs_sphinx/source/big_query.md create mode 100644 docs_sphinx/source/conf.py create mode 100644 docs_sphinx/source/csv.md create mode 100644 docs_sphinx/source/getting_started.md create mode 100644 docs_sphinx/source/grafana.md create mode 100644 docs_sphinx/source/graphite.md create mode 100644 docs_sphinx/source/index.rst create mode 100644 docs_sphinx/source/install.md create mode 100644 docs_sphinx/source/postgresql.md diff --git a/docs_sphinx/Makefile b/docs_sphinx/Makefile new file mode 100644 index 0000000..d0c3cbf --- /dev/null +++ b/docs_sphinx/Makefile @@ -0,0 +1,20 @@ +# Minimal makefile for Sphinx documentation +# + +# You can set these variables from the command line, and also +# from the environment for the first two. +SPHINXOPTS ?= +SPHINXBUILD ?= sphinx-build +SOURCEDIR = source +BUILDDIR = build + +# Put it first so that "make" without argument is like "make help". +help: + @$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) + +.PHONY: help Makefile + +# Catch-all target: route all unknown targets to Sphinx using the new +# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS). +%: Makefile + @$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) diff --git a/docs_sphinx/make.bat b/docs_sphinx/make.bat new file mode 100644 index 0000000..747ffb7 --- /dev/null +++ b/docs_sphinx/make.bat @@ -0,0 +1,35 @@ +@ECHO OFF + +pushd %~dp0 + +REM Command file for Sphinx documentation + +if "%SPHINXBUILD%" == "" ( + set SPHINXBUILD=sphinx-build +) +set SOURCEDIR=source +set BUILDDIR=build + +%SPHINXBUILD% >NUL 2>NUL +if errorlevel 9009 ( + echo. + echo.The 'sphinx-build' command was not found. Make sure you have Sphinx + echo.installed, then set the SPHINXBUILD environment variable to point + echo.to the full path of the 'sphinx-build' executable. Alternatively you + echo.may add the Sphinx directory to PATH. + echo. + echo.If you don't have Sphinx installed, grab it from + echo.https://www.sphinx-doc.org/ + exit /b 1 +) + +if "%1" == "" goto help + +%SPHINXBUILD% -M %1 %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O% +goto end + +:help +%SPHINXBUILD% -M help %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O% + +:end +popd diff --git a/docs_sphinx/source/basics.md b/docs_sphinx/source/basics.md new file mode 100644 index 0000000..77fedb4 --- /dev/null +++ b/docs_sphinx/source/basics.md @@ -0,0 +1,197 @@ + + +# Basics + +## Listing Available Tests + +``` +otava list-groups +``` + +Lists all available test groups - high-level categories of tests. + +``` +otava list-tests [group name] +``` + +Lists all tests or the tests within a given group, if the group name is provided. + +## Listing Available Metrics for Tests + +To list all available metrics defined for the test: + +``` +otava list-metrics +``` + +### Example + +> **TIP** +> See [otava.yaml](../examples/csv/config/otava.yaml) for the full example configuration. + +``` +$ otava list-metrics local.sample +metric1 +metric2 +``` + +## Finding Change Points + +``` +otava analyze ... +otava analyze ... +``` + +This command prints interesting results of all +runs of the test and a list of change-points. +A change-point is a moment when a metric value starts to differ significantly +from the values of the earlier runs and when the difference +is persistent and statistically significant that it is unlikely to happen by chance. +Otava calculates the probability (P-value) that the change point was caused +by chance - the closer to zero, the more "sure" it is about the regression or +performance improvement. The smaller is the actual magnitude of the change, +the more data points are needed to confirm the change, therefore Otava may +not notice the regression immediately after the first run that regressed. +However, it will eventually identify the specific commit that caused the regression, +as it analyzes the history of changes rather than just the HEAD of a branch. + +The `analyze` command accepts multiple tests or test groups. +The results are simply concatenated. + +### Example + +> [!TIP] +> See [otava.yaml](../examples/csv/config/otava.yaml) for the full +> example configuration and [local_samples.csv](../examples/csv/data/local_samples.csv) +> for the data. + +``` +$ otava analyze local.sample --since=2024-01-01 +INFO: Computing change points for test sample.csv... +sample: +time metric1 metric2 +------------------------- --------- --------- +2021-01-01 02:00:00 +0000 154023 10.43 +2021-01-02 02:00:00 +0000 138455 10.23 +2021-01-03 02:00:00 +0000 143112 10.29 +2021-01-04 02:00:00 +0000 149190 10.91 +2021-01-05 02:00:00 +0000 132098 10.34 +2021-01-06 02:00:00 +0000 151344 10.69 + ········· + -12.9% + ········· +2021-01-07 02:00:00 +0000 155145 9.23 +2021-01-08 02:00:00 +0000 148889 9.11 +2021-01-09 02:00:00 +0000 149466 9.13 +2021-01-10 02:00:00 +0000 148209 9.03 +``` + +## Avoiding test definition duplication + +You may find that your test definitions are very similar to each other, e.g. they all have the same metrics. Instead +of copy-pasting the definitions you can use templating capability built-in otava to define the common bits of configs +separately. + +First, extract the common pieces to the `templates` section: +```yaml +templates: + common-metrics: + throughput: + suffix: client.throughput + response-time: + suffix: client.p50 + direction: -1 # lower is better + cpu-load: + suffix: server.cpu + direction: -1 # lower is better +``` + +Next you can recall a template in the `inherit` property of the test: + +```yaml +my-product.test-1: + type: graphite + tags: [perf-test, daily, my-product, test-1] + prefix: performance-tests.daily.my-product.test-1 + inherit: common-metrics +my-product.test-2: + type: graphite + tags: [perf-test, daily, my-product, test-2] + prefix: performance-tests.daily.my-product.test-2 + inherit: common-metrics +``` + +You can inherit more than one template. + +## Validating Performance of a Feature Branch + +The `otava regressions` command can work with feature branches. + +First you need to tell Otava how to fetch the data of the tests run against a feature branch. +The `prefix` property of the graphite test definition accepts `%{BRANCH}` variable, +which is substituted at the data import time by the branch name passed to `--branch` +command argument. Alternatively, if the prefix for the main branch of your product is different +from the prefix used for feature branches, you can define an additional `branch_prefix` property. + +```yaml +my-product.test-1: + type: graphite + tags: [perf-test, daily, my-product, test-1] + prefix: performance-tests.daily.%{BRANCH}.my-product.test-1 + inherit: common-metrics + +my-product.test-2: + type: graphite + tags: [perf-test, daily, my-product, test-2] + prefix: performance-tests.daily.master.my-product.test-2 + branch_prefix: performance-tests.feature.%{BRANCH}.my-product.test-2 + inherit: common-metrics +``` + +Now you can verify if correct data are imported by running +`otava analyze --branch `. + +The `--branch` argument also works with `otava regressions`. In this case a comparison will be made +between the tail of the specified branch and the tail of the main branch (or a point of the +main branch specified by one of the `--since` selectors). + +``` +$ otava regressions --branch +$ otava regressions --branch --since +$ otava regressions --branch --since-version +$ otava regressions --branch --since-commit +``` + +When comparing two branches, you generally want to compare the tails of both test histories, and +specifically a stable sequence from the end that doesn't contain any changes in itself. +To ignore the older test results, and compare +only the last few points on the branch with the tail of the main branch, +use the `--last ` selector. E.g. to check regressions on the last run of the tests +on the feature branch: + +``` +$ otava regressions --branch --last 1 +``` + +Please beware that performance validation based on a single data point is quite weak +and Otava might miss a regression if the point is not too much different from +the baseline. However, accuracy improves as more data points accumulate, and it is +a normal way of using Otava to just merge a feature and then revert if it is +flagged later. diff --git a/docs_sphinx/source/big_query.md b/docs_sphinx/source/big_query.md new file mode 100644 index 0000000..5c0ade5 --- /dev/null +++ b/docs_sphinx/source/big_query.md @@ -0,0 +1,43 @@ + + +# BigQuery + +## Schema + +See [schema.sql](../examples/bigquery/schema.sql) for the example schema. + +## Usage + +Define BigQuery connection details via environment variables: + +```bash +export BIGQUERY_PROJECT_ID=... +export BIGQUERY_DATASET=... +export BIGQUERY_VAULT_SECRET=... +``` +or in `otava.yaml`. + +Also configure the credentials. See [config_credentials.sh](../examples/bigquery/config_credentials.sh) for an example. + +The following command shows results for a single test `aggregate_mem` and updates the database with newly found change points: + +```bash +$ BRANCH=trunk OTAVA_CONFIG=otava.yaml otava analyze aggregate_mem --update-bigquery +``` diff --git a/docs_sphinx/source/conf.py b/docs_sphinx/source/conf.py new file mode 100644 index 0000000..da5c1a0 --- /dev/null +++ b/docs_sphinx/source/conf.py @@ -0,0 +1,32 @@ +# Configuration file for the Sphinx documentation builder. +# +# For the full list of built-in configuration values, see the documentation: +# https://www.sphinx-doc.org/en/master/usage/configuration.html + +# -- Project information ----------------------------------------------------- +# https://www.sphinx-doc.org/en/master/usage/configuration.html#project-information + +project = 'otava' +copyright = '2026, Apache Otava' +author = 'Apache Otava' +release = '0.1' + +# -- General configuration --------------------------------------------------- +# https://www.sphinx-doc.org/en/master/usage/configuration.html#general-configuration + +extensions = ["myst_parser"] + +templates_path = ['_templates'] +exclude_patterns = [] + + + +# -- Options for HTML output ------------------------------------------------- +# https://www.sphinx-doc.org/en/master/usage/configuration.html#options-for-html-output + +html_theme = 'alabaster' +source_suffix = { + ".rst": "restructuredtext", + ".md": "markdown", +} +html_static_path = ['_static'] diff --git a/docs_sphinx/source/csv.md b/docs_sphinx/source/csv.md new file mode 100644 index 0000000..6d674bf --- /dev/null +++ b/docs_sphinx/source/csv.md @@ -0,0 +1,64 @@ + + +# Importing results from CSV + +> [!TIP] +> See [otava.yaml](../examples/csv/config/otava.yaml) for the full example configuration. + +## Tests + +```yaml +tests: + local.sample: + type: csv + file: tests/local_sample.csv + time_column: time + attributes: [commit] + metrics: [metric1, metric2] + csv_options: + delimiter: ',' + quotechar: "'" +``` + +## Example + +```bash +docker-compose -f examples/csv/docker-compose.yaml run --rm otava analyze local.sample +``` + +Expected output: + +```bash +time commit metric1 metric2 +------------------------- -------- --------- --------- +2024-01-01 02:00:00 +0000 aaa0 154023 10.43 +2024-01-02 02:00:00 +0000 aaa1 138455 10.23 +2024-01-03 02:00:00 +0000 aaa2 143112 10.29 +2024-01-04 02:00:00 +0000 aaa3 149190 10.91 +2024-01-05 02:00:00 +0000 aaa4 132098 10.34 +2024-01-06 02:00:00 +0000 aaa5 151344 10.69 + ········· + -12.9% + ········· +2024-01-07 02:00:00 +0000 aaa6 155145 9.23 +2024-01-08 02:00:00 +0000 aaa7 148889 9.11 +2024-01-09 02:00:00 +0000 aaa8 149466 9.13 +2024-01-10 02:00:00 +0000 aaa9 148209 9.03 +``` diff --git a/docs_sphinx/source/getting_started.md b/docs_sphinx/source/getting_started.md new file mode 100644 index 0000000..5a5b145 --- /dev/null +++ b/docs_sphinx/source/getting_started.md @@ -0,0 +1,154 @@ + + +# Getting Started + +## Installation + +Otava requires Python 3.10 or later. + +```bash +pip install apache-otava +``` + +or + +```bash +docker pull apache/otava +``` + + +## Setup + +Copy the main configuration file `resources/otava.yaml` to `~/.otava/otava.yaml` and adjust data source configuration. + +> [!TIP] +> See docs on specific data sources to learn more about their configuration - [CSV](CSV.md), [Graphite](GRAPHITE.md), +[PostgreSQL](POSTGRESQL.md), or [BigQuery](BIG_QUERY.md). + +Alternatively, it is possible to leave the config file as is, and provide credentials in the environment +by setting appropriate environment variables. +Environment variables are interpolated before interpreting the configuration file. + +## Defining tests + +All test configurations are defined in the main configuration file. +Otava supports reading data from and publishing results to a CSV file, [Graphite](https://graphiteapp.org/), +[PostgreSQL](https://www.postgresql.org/), and [BigQuery](https://cloud.google.com/bigquery). + +Tests are defined in the `tests` section. For example, the following definition will import results of the test from a CSV file: + +```yaml +tests: + local.sample: + type: csv + file: tests/resources/sample.csv + time_column: time + metrics: [metric1, metric2] + attributes: [commit] + csv_options: + delimiter: "," + quote_char: "'" +``` + +The `time_column` property points to the name of the column storing the timestamp +of each test-run. The data points will be ordered by that column. + +The `metrics` property selects the columns that hold the values to be analyzed. These values must +be numbers convertible to floats. The `metrics` property can be not only a simple list of column +names, but it can also be a dictionary configuring other properties of each metric, +the column name or direction: + +```yaml +metrics: + resp_time_p99: + direction: -1 + column: p99 +``` + +Direction can be 1 or -1. If direction is set to 1, this means that the higher the metric, the +better the performance is. If it is set to -1, higher values mean worse performance. + +The `attributes` property describes any other columns that should be attached to the final +report. Special attribute `version` and `commit` can be used to query for a given time-range. + +> [!TIP] +> To learn how to avoid repeating the same configuration in multiple tests, +> see [Avoiding test definition duplication](BASICS.md#avoiding-test-definition-duplication). + +## Listing Available Tests + +``` +otava list-groups +otava list-tests [group name] +``` + +## Listing Available Metrics for Tests + +To list all available metrics defined for the test: +``` +otava list-metrics +``` + +## Finding Change Points + +> [!TIP] +> For more details, see [Finding Change Points](BASICS.md#finding-change-points) and +> [Validating Performance of a Feature Branch](BASICS.md#validating-performance-of-a-feature-branch). + +``` +otava analyze ... +otava analyze ... +``` + +This command prints interesting results of all runs of the test and a list of change-points. + +A change-point is a moment when a metric value starts to differ significantly from the values of the earlier runs and +when the difference is statistically significant. + +Otava calculates the probability (P-value) that the change point was not caused by chance - the closer to zero, the more +certain it is about the regression or performance improvement. The smaller the magnitude of the change, the +more data points are needed to confirm the change, therefore Otava may not notice the regression immediately after the first run +that regressed. + +The `analyze` command accepts multiple tests or test groups. +The results are simply concatenated. + +## Example + +``` +$ otava analyze local.sample +INFO: Computing change points for test sample.csv... +sample: +time metric1 metric2 +------------------------- --------- --------- +2021-01-01 02:00:00 +0000 154023 10.43 +2021-01-02 02:00:00 +0000 138455 10.23 +2021-01-03 02:00:00 +0000 143112 10.29 +2021-01-04 02:00:00 +0000 149190 10.91 +2021-01-05 02:00:00 +0000 132098 10.34 +2021-01-06 02:00:00 +0000 151344 10.69 + ········· + -12.9% + ········· +2021-01-07 02:00:00 +0000 155145 9.23 +2021-01-08 02:00:00 +0000 148889 9.11 +2021-01-09 02:00:00 +0000 149466 9.13 +2021-01-10 02:00:00 +0000 148209 9.03 +``` diff --git a/docs_sphinx/source/grafana.md b/docs_sphinx/source/grafana.md new file mode 100644 index 0000000..4d5fb0d --- /dev/null +++ b/docs_sphinx/source/grafana.md @@ -0,0 +1,84 @@ + + +# Annotating Change Points in Grafana + +Change points found by `analyze` can be exported +as Grafana annotations using the `--update-grafana` flag: + +``` +$ otava analyze --update-grafana +``` + +The annotations generated by Otava get the following tags: +- `otava` +- `change-point` +- `test:` +- `metric:` +- tags configured in the `tags` property of the test +- tags configured in the `annotate` property of the test +- tags configured in the `annotate` property of the metric + +Additionally, the `annotate` property supports variable tags: +- `%{TEST_NAME}` - name of the test +- `%{METRIC_NAME}` - name of the metric +- `%{GRAPHITE_PATH}` - resolves to the path to the data in Graphite +- `%{GRAPHITE_PATH_COMPONENTS}` - splits the path of the data in Graphite into separate components + and each path component is exported as a separate tag +- `%{GRAPHITE_PREFIX}` - resolves to the prefix of the path to the data in Graphite + (the part of the path up to the metric suffix) +- `%{GRAPHITE_PREFIX_COMPONENTS}` - similar as `%{GRAPHITE_PATH_COMPONENTS}` but splits the prefix +of the path instead of the path + +## Example + +> [!TIP] +> See [otava.yaml](../examples/graphite/config/otava.yaml) for the full Graphite & Grafana example. + +Start docker-compose with Graphite in one tab: + +```bash +docker-compose -f examples/graphite/docker-compose.yaml up --force-recreate --always-recreate-deps --renew-anon-volumes +```` + + +Run otava in another tab: + +```bash +docker-compose -f examples/graphite/docker-compose.yaml run --rm otava analyze my-product.test --since=-10m --update-grafana +``` + +Expected output: + +```bash +time run branch version commit throughput response_time cpu_usage +------------------------- ----- -------- --------- -------- ------------ --------------- ----------- +2024-12-14 22:45:10 +0000 61160 87 0.2 +2024-12-14 22:46:10 +0000 60160 85 0.3 +2024-12-14 22:47:10 +0000 60960 89 0.1 + ············ ··········· + -5.6% +300.0% + ············ ··········· +2024-12-14 22:48:10 +0000 57123 88 0.8 +2024-12-14 22:49:10 +0000 57980 87 0.9 +2024-12-14 22:50:10 +0000 56950 85 0.7 +``` + +Open local [Grafana](http://localhost:3000) in your browser (use default `admin/admin` credentials), open +`Benchmarks` dashboard, and see your data points and annotations on both charts. diff --git a/docs_sphinx/source/graphite.md b/docs_sphinx/source/graphite.md new file mode 100644 index 0000000..2f01129 --- /dev/null +++ b/docs_sphinx/source/graphite.md @@ -0,0 +1,125 @@ + + +# Importing results from Graphite + +> [!TIP] +> See [otava.yaml](../examples/graphite/config/otava.yaml) for the full example configuration. + +## Graphite and Grafana Connection + +The following block contains Graphite and Grafana connection details: + +```yaml +graphite: + url: ... + +grafana: + url: ... + user: ... + password: ... +``` + +These variables can be specified directly in `otava.yaml` or passed as environment variables: + +```yaml +graphite: + url: ${GRAPHITE_ADDRESS} + +grafana: + url: ${GRAFANA_ADDRESS} + user: ${GRAFANA_USER} + password: ${GRAFANA_PASSWORD} +``` + + +## Tests + +### Importing results from Graphite + +Test configuration contains queries selecting experiment data from Graphite. This is done by specifying the Graphite +path prefix common for all the test's metrics and suffixes for each of the metrics recorded by the test run. + +```yaml +tests: + my-product.test: + type: graphite + prefix: performance-tests.daily.my-product + metrics: + throughput: + suffix: client.throughput + response-time: + suffix: client.p50 + direction: -1 # lower is better + cpu-load: + suffix: server.cpu + direction: -1 # lower is better +``` + +### Tags + +> [!WARNING] +> Tags do not work as expected in the current version. See https://github.com/apache/otava/issues/24 for more details + +The optional `tags` property contains the tags that are used to query for Graphite events that store +additional test run metadata such as run identifier, commit, branch and product version information. + +The following command will post an event with the test run metadata: +```shell +$ curl -X POST "http://graphite_address/events/" \ + -d '{ + "what": "Performance Test", + "tags": ["perf-test", "daily", "my-product"], + "when": 1537884100, + "data": {"commit": "fe6583ab", "branch": "new-feature", "version": "0.0.1"} + }' +``` + +Posting those events is not mandatory, but when they are available, Otava is able to +filter data by commit or version using `--since-commit` or `--since-version` selectors. + +## Example + +Start docker-compose with Graphite in one tab: + +```bash +docker-compose -f examples/graphite/docker-compose.yaml up --force-recreate --always-recreate-deps --renew-anon-volumes +```` + +Run otava in another tab: + +```bash +docker-compose -f examples/graphite/docker-compose.yaml run --rm otava analyze my-product.test --since=-10m +``` + +Expected output: + +```bash +time run branch version commit throughput response_time cpu_usage +------------------------- ----- -------- --------- -------- ------------ --------------- ----------- +2024-12-14 22:45:10 +0000 61160 87 0.2 +2024-12-14 22:46:10 +0000 60160 85 0.3 +2024-12-14 22:47:10 +0000 60960 89 0.1 + ············ ··········· + -5.6% +300.0% + ············ ··········· +2024-12-14 22:48:10 +0000 57123 88 0.8 +2024-12-14 22:49:10 +0000 57980 87 0.9 +2024-12-14 22:50:10 +0000 56950 85 0.7 +``` diff --git a/docs_sphinx/source/index.rst b/docs_sphinx/source/index.rst new file mode 100644 index 0000000..909f3b6 --- /dev/null +++ b/docs_sphinx/source/index.rst @@ -0,0 +1,17 @@ +Welcome to Apache Otava Documentation +===================================== + +This documentation is powered by Sphinx and supports Markdown via MyST. + +.. toctree:: + :maxdepth: 2 + :caption: Documentation: + + getting_started + install + basics + csv + big_query + postgresql + grafana + graphite diff --git a/docs_sphinx/source/install.md b/docs_sphinx/source/install.md new file mode 100644 index 0000000..1844f89 --- /dev/null +++ b/docs_sphinx/source/install.md @@ -0,0 +1,36 @@ + + +# Installation + +## Install using pip + +Otava requires Python 3.10 or later. + +```bash +pip install apache-otava +``` + +## Install using Docker + +Pull the official Docker image: + +```bash +docker pull apache/otava +``` diff --git a/docs_sphinx/source/postgresql.md b/docs_sphinx/source/postgresql.md new file mode 100644 index 0000000..870aa5d --- /dev/null +++ b/docs_sphinx/source/postgresql.md @@ -0,0 +1,138 @@ + + +# Importing results from PostgreSQL + +> [!TIP] +> See [otava.yaml](../examples/postgresql/config/otava.yaml) for the full example configuration. + +## PostgreSQL Connection +The following block contains PostgreSQL connection details: + +```yaml +postgres: + hostname: ... + port: ... + username: ... + password: ... + database: ... +``` + +These variables can be specified directly in `otava.yaml` or passed as environment variables: + +```yaml +postgres: + hostname: ${POSTGRES_HOSTNAME} + port: ${POSTGRES_PORT} + username: ${POSTGRES_USERNAME} + password: ${POSTGRES_PASSWORD} + database: ${POSTGRES_DATABASE} +``` + +## Tests + +Test configuration contains queries selecting experiment data, a time column, and a list of columns to analyze: + +```yaml +tests: + aggregate_mem: + type: postgres + time_column: commit_ts + attributes: [experiment_id, config_id, commit] + metrics: + process_cumulative_rate_mean: + direction: 1 + scale: 1 + process_cumulative_rate_stderr: + direction: -1 + scale: 1 + process_cumulative_rate_diff: + direction: -1 + scale: 1 + query: | + SELECT e.commit, + e.commit_ts, + r.process_cumulative_rate_mean, + r.process_cumulative_rate_stderr, + r.process_cumulative_rate_diff, + r.experiment_id, + r.config_id + FROM results r + INNER JOIN configs c ON r.config_id = c.id + INNER JOIN experiments e ON r.experiment_id = e.id + WHERE e.exclude_from_analysis = false AND + e.branch = 'trunk' AND + e.username = 'ci' AND + c.store = 'MEM' AND + c.cache = true AND + c.benchmark = 'aggregate' AND + c.instance_type = 'ec2i3.large' + ORDER BY e.commit_ts ASC; +``` + +## Example + +### Usage + +Start docker-compose with PostgreSQL in one tab: + +```bash +docker-compose -f examples/postgresql/docker-compose.yaml up --force-recreate --always-recreate-deps --renew-anon-volumes +```` + +Run Otava in the other tab to show results for a single test `aggregate_mem` and update the database with newly found change points: + +```bash +docker-compose -f examples/postgresql/docker-compose.yaml run --rm otava analyze aggregate_mem --update-postgres +``` + +Expected output: + +```bash 0.0s +time experiment_id commit process_cumulative_rate_mean process_cumulative_rate_stderr process_cumulative_rate_diff +------------------------- ------------------ -------- ------------------------------ -------------------------------- ------------------------------ +2024-03-13 10:03:02 +0000 aggregate-36e5ccd2 36e5ccd2 61160 2052 13558 +2024-03-25 10:03:02 +0000 aggregate-d5460f38 d5460f38 60160 2142 13454 +2024-04-02 10:03:02 +0000 aggregate-bc9425cb bc9425cb 60960 2052 13053 + ······························ + -5.6% + ······························ +2024-04-06 10:03:02 +0000 aggregate-14df1b11 14df1b11 57123 2052 14052 +2024-04-13 10:03:02 +0000 aggregate-ac40c0d8 ac40c0d8 57980 2052 13521 +2024-04-27 10:03:02 +0000 aggregate-0af4ccbc 0af4ccbc 56950 2052 13532 +``` + +### Configuration + +See [otava.yaml](../examples/postgresql/config/otava.yaml) for the example configuration: +* Block `postgres` contains connection details to the PostgreSQL database. +* Block `templates` contains common pieces of configuration used by all tests - time column and a list of attributes and metrics. +* Block `tests` contains configuration for the individual tests, specifically a query that fetches analyzed columns sorted by commit timestamp. + +[schema.sql](../examples/postgresql/init-db/schema.sql) contains the schema used in this example. + +[docker-compose.yaml](../examples/postgresql/docker-compose.yaml) contains example config required to connect to PostgreSQL: +1. `POSTGRES_*` environment variables are used to pass connection details to the container. +2. `OTAVA_CONFIG` is the path to the configuration file described above. +3. `BRANCH` variable is used within `OTAVA_CONFIG` to analyze experiment results only for a specific branch. + + +### CLI arguments + +* `--update-postgres` - updates the database with newly found change points. From 9241b0743e08d729982c99a6577f0c2689c1583a Mon Sep 17 00:00:00 2001 From: Phanindra899 Date: Tue, 31 Mar 2026 03:56:12 +0530 Subject: [PATCH 4/6] Ignore Sphinx build and virtual environment files --- .gitignore | 2 ++ 1 file changed, 2 insertions(+) diff --git a/.gitignore b/.gitignore index 7b70f42..835ee8f 100644 --- a/.gitignore +++ b/.gitignore @@ -31,3 +31,5 @@ dist/ venv/ .tox/ .docker/ +docs_sphinx/venv/ +docs_sphinx/build/ From 68479e92f41227cd4f02ea8ee7afe70b729ad8d5 Mon Sep 17 00:00:00 2001 From: Phanindra899 Date: Sat, 4 Apr 2026 16:31:28 +0530 Subject: [PATCH 5/6] Fix Sphinx index to use docs/ as source --- docs_sphinx/source/basics.md | 197 -------------------------- docs_sphinx/source/big_query.md | 43 ------ docs_sphinx/source/csv.md | 64 --------- docs_sphinx/source/getting_started.md | 154 -------------------- docs_sphinx/source/grafana.md | 84 ----------- docs_sphinx/source/graphite.md | 125 ---------------- docs_sphinx/source/index.rst | 25 ++-- docs_sphinx/source/install.md | 36 ----- docs_sphinx/source/postgresql.md | 138 ------------------ 9 files changed, 13 insertions(+), 853 deletions(-) delete mode 100644 docs_sphinx/source/basics.md delete mode 100644 docs_sphinx/source/big_query.md delete mode 100644 docs_sphinx/source/csv.md delete mode 100644 docs_sphinx/source/getting_started.md delete mode 100644 docs_sphinx/source/grafana.md delete mode 100644 docs_sphinx/source/graphite.md delete mode 100644 docs_sphinx/source/install.md delete mode 100644 docs_sphinx/source/postgresql.md diff --git a/docs_sphinx/source/basics.md b/docs_sphinx/source/basics.md deleted file mode 100644 index 77fedb4..0000000 --- a/docs_sphinx/source/basics.md +++ /dev/null @@ -1,197 +0,0 @@ - - -# Basics - -## Listing Available Tests - -``` -otava list-groups -``` - -Lists all available test groups - high-level categories of tests. - -``` -otava list-tests [group name] -``` - -Lists all tests or the tests within a given group, if the group name is provided. - -## Listing Available Metrics for Tests - -To list all available metrics defined for the test: - -``` -otava list-metrics -``` - -### Example - -> **TIP** -> See [otava.yaml](../examples/csv/config/otava.yaml) for the full example configuration. - -``` -$ otava list-metrics local.sample -metric1 -metric2 -``` - -## Finding Change Points - -``` -otava analyze ... -otava analyze ... -``` - -This command prints interesting results of all -runs of the test and a list of change-points. -A change-point is a moment when a metric value starts to differ significantly -from the values of the earlier runs and when the difference -is persistent and statistically significant that it is unlikely to happen by chance. -Otava calculates the probability (P-value) that the change point was caused -by chance - the closer to zero, the more "sure" it is about the regression or -performance improvement. The smaller is the actual magnitude of the change, -the more data points are needed to confirm the change, therefore Otava may -not notice the regression immediately after the first run that regressed. -However, it will eventually identify the specific commit that caused the regression, -as it analyzes the history of changes rather than just the HEAD of a branch. - -The `analyze` command accepts multiple tests or test groups. -The results are simply concatenated. - -### Example - -> [!TIP] -> See [otava.yaml](../examples/csv/config/otava.yaml) for the full -> example configuration and [local_samples.csv](../examples/csv/data/local_samples.csv) -> for the data. - -``` -$ otava analyze local.sample --since=2024-01-01 -INFO: Computing change points for test sample.csv... -sample: -time metric1 metric2 -------------------------- --------- --------- -2021-01-01 02:00:00 +0000 154023 10.43 -2021-01-02 02:00:00 +0000 138455 10.23 -2021-01-03 02:00:00 +0000 143112 10.29 -2021-01-04 02:00:00 +0000 149190 10.91 -2021-01-05 02:00:00 +0000 132098 10.34 -2021-01-06 02:00:00 +0000 151344 10.69 - ········· - -12.9% - ········· -2021-01-07 02:00:00 +0000 155145 9.23 -2021-01-08 02:00:00 +0000 148889 9.11 -2021-01-09 02:00:00 +0000 149466 9.13 -2021-01-10 02:00:00 +0000 148209 9.03 -``` - -## Avoiding test definition duplication - -You may find that your test definitions are very similar to each other, e.g. they all have the same metrics. Instead -of copy-pasting the definitions you can use templating capability built-in otava to define the common bits of configs -separately. - -First, extract the common pieces to the `templates` section: -```yaml -templates: - common-metrics: - throughput: - suffix: client.throughput - response-time: - suffix: client.p50 - direction: -1 # lower is better - cpu-load: - suffix: server.cpu - direction: -1 # lower is better -``` - -Next you can recall a template in the `inherit` property of the test: - -```yaml -my-product.test-1: - type: graphite - tags: [perf-test, daily, my-product, test-1] - prefix: performance-tests.daily.my-product.test-1 - inherit: common-metrics -my-product.test-2: - type: graphite - tags: [perf-test, daily, my-product, test-2] - prefix: performance-tests.daily.my-product.test-2 - inherit: common-metrics -``` - -You can inherit more than one template. - -## Validating Performance of a Feature Branch - -The `otava regressions` command can work with feature branches. - -First you need to tell Otava how to fetch the data of the tests run against a feature branch. -The `prefix` property of the graphite test definition accepts `%{BRANCH}` variable, -which is substituted at the data import time by the branch name passed to `--branch` -command argument. Alternatively, if the prefix for the main branch of your product is different -from the prefix used for feature branches, you can define an additional `branch_prefix` property. - -```yaml -my-product.test-1: - type: graphite - tags: [perf-test, daily, my-product, test-1] - prefix: performance-tests.daily.%{BRANCH}.my-product.test-1 - inherit: common-metrics - -my-product.test-2: - type: graphite - tags: [perf-test, daily, my-product, test-2] - prefix: performance-tests.daily.master.my-product.test-2 - branch_prefix: performance-tests.feature.%{BRANCH}.my-product.test-2 - inherit: common-metrics -``` - -Now you can verify if correct data are imported by running -`otava analyze --branch `. - -The `--branch` argument also works with `otava regressions`. In this case a comparison will be made -between the tail of the specified branch and the tail of the main branch (or a point of the -main branch specified by one of the `--since` selectors). - -``` -$ otava regressions --branch -$ otava regressions --branch --since -$ otava regressions --branch --since-version -$ otava regressions --branch --since-commit -``` - -When comparing two branches, you generally want to compare the tails of both test histories, and -specifically a stable sequence from the end that doesn't contain any changes in itself. -To ignore the older test results, and compare -only the last few points on the branch with the tail of the main branch, -use the `--last ` selector. E.g. to check regressions on the last run of the tests -on the feature branch: - -``` -$ otava regressions --branch --last 1 -``` - -Please beware that performance validation based on a single data point is quite weak -and Otava might miss a regression if the point is not too much different from -the baseline. However, accuracy improves as more data points accumulate, and it is -a normal way of using Otava to just merge a feature and then revert if it is -flagged later. diff --git a/docs_sphinx/source/big_query.md b/docs_sphinx/source/big_query.md deleted file mode 100644 index 5c0ade5..0000000 --- a/docs_sphinx/source/big_query.md +++ /dev/null @@ -1,43 +0,0 @@ - - -# BigQuery - -## Schema - -See [schema.sql](../examples/bigquery/schema.sql) for the example schema. - -## Usage - -Define BigQuery connection details via environment variables: - -```bash -export BIGQUERY_PROJECT_ID=... -export BIGQUERY_DATASET=... -export BIGQUERY_VAULT_SECRET=... -``` -or in `otava.yaml`. - -Also configure the credentials. See [config_credentials.sh](../examples/bigquery/config_credentials.sh) for an example. - -The following command shows results for a single test `aggregate_mem` and updates the database with newly found change points: - -```bash -$ BRANCH=trunk OTAVA_CONFIG=otava.yaml otava analyze aggregate_mem --update-bigquery -``` diff --git a/docs_sphinx/source/csv.md b/docs_sphinx/source/csv.md deleted file mode 100644 index 6d674bf..0000000 --- a/docs_sphinx/source/csv.md +++ /dev/null @@ -1,64 +0,0 @@ - - -# Importing results from CSV - -> [!TIP] -> See [otava.yaml](../examples/csv/config/otava.yaml) for the full example configuration. - -## Tests - -```yaml -tests: - local.sample: - type: csv - file: tests/local_sample.csv - time_column: time - attributes: [commit] - metrics: [metric1, metric2] - csv_options: - delimiter: ',' - quotechar: "'" -``` - -## Example - -```bash -docker-compose -f examples/csv/docker-compose.yaml run --rm otava analyze local.sample -``` - -Expected output: - -```bash -time commit metric1 metric2 -------------------------- -------- --------- --------- -2024-01-01 02:00:00 +0000 aaa0 154023 10.43 -2024-01-02 02:00:00 +0000 aaa1 138455 10.23 -2024-01-03 02:00:00 +0000 aaa2 143112 10.29 -2024-01-04 02:00:00 +0000 aaa3 149190 10.91 -2024-01-05 02:00:00 +0000 aaa4 132098 10.34 -2024-01-06 02:00:00 +0000 aaa5 151344 10.69 - ········· - -12.9% - ········· -2024-01-07 02:00:00 +0000 aaa6 155145 9.23 -2024-01-08 02:00:00 +0000 aaa7 148889 9.11 -2024-01-09 02:00:00 +0000 aaa8 149466 9.13 -2024-01-10 02:00:00 +0000 aaa9 148209 9.03 -``` diff --git a/docs_sphinx/source/getting_started.md b/docs_sphinx/source/getting_started.md deleted file mode 100644 index 5a5b145..0000000 --- a/docs_sphinx/source/getting_started.md +++ /dev/null @@ -1,154 +0,0 @@ - - -# Getting Started - -## Installation - -Otava requires Python 3.10 or later. - -```bash -pip install apache-otava -``` - -or - -```bash -docker pull apache/otava -``` - - -## Setup - -Copy the main configuration file `resources/otava.yaml` to `~/.otava/otava.yaml` and adjust data source configuration. - -> [!TIP] -> See docs on specific data sources to learn more about their configuration - [CSV](CSV.md), [Graphite](GRAPHITE.md), -[PostgreSQL](POSTGRESQL.md), or [BigQuery](BIG_QUERY.md). - -Alternatively, it is possible to leave the config file as is, and provide credentials in the environment -by setting appropriate environment variables. -Environment variables are interpolated before interpreting the configuration file. - -## Defining tests - -All test configurations are defined in the main configuration file. -Otava supports reading data from and publishing results to a CSV file, [Graphite](https://graphiteapp.org/), -[PostgreSQL](https://www.postgresql.org/), and [BigQuery](https://cloud.google.com/bigquery). - -Tests are defined in the `tests` section. For example, the following definition will import results of the test from a CSV file: - -```yaml -tests: - local.sample: - type: csv - file: tests/resources/sample.csv - time_column: time - metrics: [metric1, metric2] - attributes: [commit] - csv_options: - delimiter: "," - quote_char: "'" -``` - -The `time_column` property points to the name of the column storing the timestamp -of each test-run. The data points will be ordered by that column. - -The `metrics` property selects the columns that hold the values to be analyzed. These values must -be numbers convertible to floats. The `metrics` property can be not only a simple list of column -names, but it can also be a dictionary configuring other properties of each metric, -the column name or direction: - -```yaml -metrics: - resp_time_p99: - direction: -1 - column: p99 -``` - -Direction can be 1 or -1. If direction is set to 1, this means that the higher the metric, the -better the performance is. If it is set to -1, higher values mean worse performance. - -The `attributes` property describes any other columns that should be attached to the final -report. Special attribute `version` and `commit` can be used to query for a given time-range. - -> [!TIP] -> To learn how to avoid repeating the same configuration in multiple tests, -> see [Avoiding test definition duplication](BASICS.md#avoiding-test-definition-duplication). - -## Listing Available Tests - -``` -otava list-groups -otava list-tests [group name] -``` - -## Listing Available Metrics for Tests - -To list all available metrics defined for the test: -``` -otava list-metrics -``` - -## Finding Change Points - -> [!TIP] -> For more details, see [Finding Change Points](BASICS.md#finding-change-points) and -> [Validating Performance of a Feature Branch](BASICS.md#validating-performance-of-a-feature-branch). - -``` -otava analyze ... -otava analyze ... -``` - -This command prints interesting results of all runs of the test and a list of change-points. - -A change-point is a moment when a metric value starts to differ significantly from the values of the earlier runs and -when the difference is statistically significant. - -Otava calculates the probability (P-value) that the change point was not caused by chance - the closer to zero, the more -certain it is about the regression or performance improvement. The smaller the magnitude of the change, the -more data points are needed to confirm the change, therefore Otava may not notice the regression immediately after the first run -that regressed. - -The `analyze` command accepts multiple tests or test groups. -The results are simply concatenated. - -## Example - -``` -$ otava analyze local.sample -INFO: Computing change points for test sample.csv... -sample: -time metric1 metric2 -------------------------- --------- --------- -2021-01-01 02:00:00 +0000 154023 10.43 -2021-01-02 02:00:00 +0000 138455 10.23 -2021-01-03 02:00:00 +0000 143112 10.29 -2021-01-04 02:00:00 +0000 149190 10.91 -2021-01-05 02:00:00 +0000 132098 10.34 -2021-01-06 02:00:00 +0000 151344 10.69 - ········· - -12.9% - ········· -2021-01-07 02:00:00 +0000 155145 9.23 -2021-01-08 02:00:00 +0000 148889 9.11 -2021-01-09 02:00:00 +0000 149466 9.13 -2021-01-10 02:00:00 +0000 148209 9.03 -``` diff --git a/docs_sphinx/source/grafana.md b/docs_sphinx/source/grafana.md deleted file mode 100644 index 4d5fb0d..0000000 --- a/docs_sphinx/source/grafana.md +++ /dev/null @@ -1,84 +0,0 @@ - - -# Annotating Change Points in Grafana - -Change points found by `analyze` can be exported -as Grafana annotations using the `--update-grafana` flag: - -``` -$ otava analyze --update-grafana -``` - -The annotations generated by Otava get the following tags: -- `otava` -- `change-point` -- `test:` -- `metric:` -- tags configured in the `tags` property of the test -- tags configured in the `annotate` property of the test -- tags configured in the `annotate` property of the metric - -Additionally, the `annotate` property supports variable tags: -- `%{TEST_NAME}` - name of the test -- `%{METRIC_NAME}` - name of the metric -- `%{GRAPHITE_PATH}` - resolves to the path to the data in Graphite -- `%{GRAPHITE_PATH_COMPONENTS}` - splits the path of the data in Graphite into separate components - and each path component is exported as a separate tag -- `%{GRAPHITE_PREFIX}` - resolves to the prefix of the path to the data in Graphite - (the part of the path up to the metric suffix) -- `%{GRAPHITE_PREFIX_COMPONENTS}` - similar as `%{GRAPHITE_PATH_COMPONENTS}` but splits the prefix -of the path instead of the path - -## Example - -> [!TIP] -> See [otava.yaml](../examples/graphite/config/otava.yaml) for the full Graphite & Grafana example. - -Start docker-compose with Graphite in one tab: - -```bash -docker-compose -f examples/graphite/docker-compose.yaml up --force-recreate --always-recreate-deps --renew-anon-volumes -```` - - -Run otava in another tab: - -```bash -docker-compose -f examples/graphite/docker-compose.yaml run --rm otava analyze my-product.test --since=-10m --update-grafana -``` - -Expected output: - -```bash -time run branch version commit throughput response_time cpu_usage -------------------------- ----- -------- --------- -------- ------------ --------------- ----------- -2024-12-14 22:45:10 +0000 61160 87 0.2 -2024-12-14 22:46:10 +0000 60160 85 0.3 -2024-12-14 22:47:10 +0000 60960 89 0.1 - ············ ··········· - -5.6% +300.0% - ············ ··········· -2024-12-14 22:48:10 +0000 57123 88 0.8 -2024-12-14 22:49:10 +0000 57980 87 0.9 -2024-12-14 22:50:10 +0000 56950 85 0.7 -``` - -Open local [Grafana](http://localhost:3000) in your browser (use default `admin/admin` credentials), open -`Benchmarks` dashboard, and see your data points and annotations on both charts. diff --git a/docs_sphinx/source/graphite.md b/docs_sphinx/source/graphite.md deleted file mode 100644 index 2f01129..0000000 --- a/docs_sphinx/source/graphite.md +++ /dev/null @@ -1,125 +0,0 @@ - - -# Importing results from Graphite - -> [!TIP] -> See [otava.yaml](../examples/graphite/config/otava.yaml) for the full example configuration. - -## Graphite and Grafana Connection - -The following block contains Graphite and Grafana connection details: - -```yaml -graphite: - url: ... - -grafana: - url: ... - user: ... - password: ... -``` - -These variables can be specified directly in `otava.yaml` or passed as environment variables: - -```yaml -graphite: - url: ${GRAPHITE_ADDRESS} - -grafana: - url: ${GRAFANA_ADDRESS} - user: ${GRAFANA_USER} - password: ${GRAFANA_PASSWORD} -``` - - -## Tests - -### Importing results from Graphite - -Test configuration contains queries selecting experiment data from Graphite. This is done by specifying the Graphite -path prefix common for all the test's metrics and suffixes for each of the metrics recorded by the test run. - -```yaml -tests: - my-product.test: - type: graphite - prefix: performance-tests.daily.my-product - metrics: - throughput: - suffix: client.throughput - response-time: - suffix: client.p50 - direction: -1 # lower is better - cpu-load: - suffix: server.cpu - direction: -1 # lower is better -``` - -### Tags - -> [!WARNING] -> Tags do not work as expected in the current version. See https://github.com/apache/otava/issues/24 for more details - -The optional `tags` property contains the tags that are used to query for Graphite events that store -additional test run metadata such as run identifier, commit, branch and product version information. - -The following command will post an event with the test run metadata: -```shell -$ curl -X POST "http://graphite_address/events/" \ - -d '{ - "what": "Performance Test", - "tags": ["perf-test", "daily", "my-product"], - "when": 1537884100, - "data": {"commit": "fe6583ab", "branch": "new-feature", "version": "0.0.1"} - }' -``` - -Posting those events is not mandatory, but when they are available, Otava is able to -filter data by commit or version using `--since-commit` or `--since-version` selectors. - -## Example - -Start docker-compose with Graphite in one tab: - -```bash -docker-compose -f examples/graphite/docker-compose.yaml up --force-recreate --always-recreate-deps --renew-anon-volumes -```` - -Run otava in another tab: - -```bash -docker-compose -f examples/graphite/docker-compose.yaml run --rm otava analyze my-product.test --since=-10m -``` - -Expected output: - -```bash -time run branch version commit throughput response_time cpu_usage -------------------------- ----- -------- --------- -------- ------------ --------------- ----------- -2024-12-14 22:45:10 +0000 61160 87 0.2 -2024-12-14 22:46:10 +0000 60160 85 0.3 -2024-12-14 22:47:10 +0000 60960 89 0.1 - ············ ··········· - -5.6% +300.0% - ············ ··········· -2024-12-14 22:48:10 +0000 57123 88 0.8 -2024-12-14 22:49:10 +0000 57980 87 0.9 -2024-12-14 22:50:10 +0000 56950 85 0.7 -``` diff --git a/docs_sphinx/source/index.rst b/docs_sphinx/source/index.rst index 909f3b6..b49957b 100644 --- a/docs_sphinx/source/index.rst +++ b/docs_sphinx/source/index.rst @@ -1,17 +1,18 @@ Welcome to Apache Otava Documentation -===================================== - -This documentation is powered by Sphinx and supports Markdown via MyST. +==================================== .. toctree:: :maxdepth: 2 - :caption: Documentation: - getting_started - install - basics - csv - big_query - postgresql - grafana - graphite + ../../docs/README.md + ../../docs/INSTALL.md + ../../docs/GETTING_STARTED.md + ../../docs/BASICS.md + ../../docs/CSV.md + ../../docs/BIG_QUERY.md + ../../docs/POSTGRESQL.md + ../../docs/GRAFANA.md + ../../docs/GRAPHITE.md + + +This documentation is powered by Sphinx and supports Markdown via MyST. diff --git a/docs_sphinx/source/install.md b/docs_sphinx/source/install.md deleted file mode 100644 index 1844f89..0000000 --- a/docs_sphinx/source/install.md +++ /dev/null @@ -1,36 +0,0 @@ - - -# Installation - -## Install using pip - -Otava requires Python 3.10 or later. - -```bash -pip install apache-otava -``` - -## Install using Docker - -Pull the official Docker image: - -```bash -docker pull apache/otava -``` diff --git a/docs_sphinx/source/postgresql.md b/docs_sphinx/source/postgresql.md deleted file mode 100644 index 870aa5d..0000000 --- a/docs_sphinx/source/postgresql.md +++ /dev/null @@ -1,138 +0,0 @@ - - -# Importing results from PostgreSQL - -> [!TIP] -> See [otava.yaml](../examples/postgresql/config/otava.yaml) for the full example configuration. - -## PostgreSQL Connection -The following block contains PostgreSQL connection details: - -```yaml -postgres: - hostname: ... - port: ... - username: ... - password: ... - database: ... -``` - -These variables can be specified directly in `otava.yaml` or passed as environment variables: - -```yaml -postgres: - hostname: ${POSTGRES_HOSTNAME} - port: ${POSTGRES_PORT} - username: ${POSTGRES_USERNAME} - password: ${POSTGRES_PASSWORD} - database: ${POSTGRES_DATABASE} -``` - -## Tests - -Test configuration contains queries selecting experiment data, a time column, and a list of columns to analyze: - -```yaml -tests: - aggregate_mem: - type: postgres - time_column: commit_ts - attributes: [experiment_id, config_id, commit] - metrics: - process_cumulative_rate_mean: - direction: 1 - scale: 1 - process_cumulative_rate_stderr: - direction: -1 - scale: 1 - process_cumulative_rate_diff: - direction: -1 - scale: 1 - query: | - SELECT e.commit, - e.commit_ts, - r.process_cumulative_rate_mean, - r.process_cumulative_rate_stderr, - r.process_cumulative_rate_diff, - r.experiment_id, - r.config_id - FROM results r - INNER JOIN configs c ON r.config_id = c.id - INNER JOIN experiments e ON r.experiment_id = e.id - WHERE e.exclude_from_analysis = false AND - e.branch = 'trunk' AND - e.username = 'ci' AND - c.store = 'MEM' AND - c.cache = true AND - c.benchmark = 'aggregate' AND - c.instance_type = 'ec2i3.large' - ORDER BY e.commit_ts ASC; -``` - -## Example - -### Usage - -Start docker-compose with PostgreSQL in one tab: - -```bash -docker-compose -f examples/postgresql/docker-compose.yaml up --force-recreate --always-recreate-deps --renew-anon-volumes -```` - -Run Otava in the other tab to show results for a single test `aggregate_mem` and update the database with newly found change points: - -```bash -docker-compose -f examples/postgresql/docker-compose.yaml run --rm otava analyze aggregate_mem --update-postgres -``` - -Expected output: - -```bash 0.0s -time experiment_id commit process_cumulative_rate_mean process_cumulative_rate_stderr process_cumulative_rate_diff -------------------------- ------------------ -------- ------------------------------ -------------------------------- ------------------------------ -2024-03-13 10:03:02 +0000 aggregate-36e5ccd2 36e5ccd2 61160 2052 13558 -2024-03-25 10:03:02 +0000 aggregate-d5460f38 d5460f38 60160 2142 13454 -2024-04-02 10:03:02 +0000 aggregate-bc9425cb bc9425cb 60960 2052 13053 - ······························ - -5.6% - ······························ -2024-04-06 10:03:02 +0000 aggregate-14df1b11 14df1b11 57123 2052 14052 -2024-04-13 10:03:02 +0000 aggregate-ac40c0d8 ac40c0d8 57980 2052 13521 -2024-04-27 10:03:02 +0000 aggregate-0af4ccbc 0af4ccbc 56950 2052 13532 -``` - -### Configuration - -See [otava.yaml](../examples/postgresql/config/otava.yaml) for the example configuration: -* Block `postgres` contains connection details to the PostgreSQL database. -* Block `templates` contains common pieces of configuration used by all tests - time column and a list of attributes and metrics. -* Block `tests` contains configuration for the individual tests, specifically a query that fetches analyzed columns sorted by commit timestamp. - -[schema.sql](../examples/postgresql/init-db/schema.sql) contains the schema used in this example. - -[docker-compose.yaml](../examples/postgresql/docker-compose.yaml) contains example config required to connect to PostgreSQL: -1. `POSTGRES_*` environment variables are used to pass connection details to the container. -2. `OTAVA_CONFIG` is the path to the configuration file described above. -3. `BRANCH` variable is used within `OTAVA_CONFIG` to analyze experiment results only for a specific branch. - - -### CLI arguments - -* `--update-postgres` - updates the database with newly found change points. From 1f550a4b78dc1f5b76dde06f007ca94a8897b6b1 Mon Sep 17 00:00:00 2001 From: Phanindra899 Date: Sat, 4 Apr 2026 16:32:26 +0530 Subject: [PATCH 6/6] Clean Sphinx config and add docs path --- docs_sphinx/source/conf.py | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/docs_sphinx/source/conf.py b/docs_sphinx/source/conf.py index da5c1a0..4d9cc2a 100644 --- a/docs_sphinx/source/conf.py +++ b/docs_sphinx/source/conf.py @@ -1,5 +1,9 @@ # Configuration file for the Sphinx documentation builder. -# +import os +import sys + +sys.path.insert(0, os.path.abspath('../..')) + # For the full list of built-in configuration values, see the documentation: # https://www.sphinx-doc.org/en/master/usage/configuration.html