Test that benchmark comparison works as intended

[gruuya]

No description provided.

[gruuya]

/benchmark

[gruuya]

/benchmark

[github-actions]

Benchmark results

Benchmarks comparing b00672a and PR 1

Comparing main-b00672a and pr-1
--------------------
Benchmark tpch.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query        ┃ main-b00672a ┃      pr-1 ┃        Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 1     │     439.71ms │ 1438.69ms │  3.27x slower │
│ QQuery 2     │      58.48ms │ 1062.12ms │ 18.16x slower │
│ QQuery 3     │     144.53ms │ 1150.29ms │  7.96x slower │
│ QQuery 4     │      86.39ms │ 1090.41ms │ 12.62x slower │
│ QQuery 5     │     196.61ms │ 1206.59ms │  6.14x slower │
│ QQuery 6     │     105.18ms │  106.39ms │     no change │
│ QQuery 7     │     279.02ms │ 1289.73ms │  4.62x slower │
│ QQuery 8     │     200.17ms │ 1203.41ms │  6.01x slower │
│ QQuery 9     │     298.34ms │ 1299.96ms │  4.36x slower │
│ QQuery 10    │     237.11ms │ 1246.28ms │  5.26x slower │
│ QQuery 11    │      62.46ms │ 1066.80ms │ 17.08x slower │
│ QQuery 12    │     124.41ms │ 1131.82ms │  9.10x slower │
│ QQuery 13    │     176.44ms │ 1182.42ms │  6.70x slower │
│ QQuery 14    │     128.97ms │  131.41ms │     no change │
│ QQuery 15    │     193.30ms │ 1191.67ms │  6.16x slower │
│ QQuery 16    │      52.41ms │ 1058.02ms │ 20.19x slower │
│ QQuery 17    │     303.16ms │  291.13ms │     no change │
│ QQuery 18    │     538.75ms │ 1440.16ms │  2.67x slower │
│ QQuery 19    │     231.66ms │  235.50ms │     no change │
│ QQuery 20    │     191.02ms │ 1201.58ms │  6.29x slower │
│ QQuery 21    │     325.14ms │ 1328.94ms │  4.09x slower │
│ QQuery 22    │      54.60ms │ 1055.05ms │ 19.32x slower │
└──────────────┴──────────────┴───────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary           ┃            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (main-b00672a)   │  4427.83ms │
│ Total Time (pr-1)           │ 22408.38ms │
│ Average Time (main-b00672a) │   201.27ms │
│ Average Time (pr-1)         │  1018.56ms │
│ Queries Faster              │          0 │
│ Queries Slower              │         18 │
│ Queries with No Change      │          4 │
└─────────────────────────────┴────────────┘

[gruuya]

Doing another benchmark test, now after removing the fake slowdown in 1a5f166.

/benchmark

[github-actions]

Benchmark results

Benchmarks comparing f296e02 (main) and 1a5f166 (PR)

Comparing f296e02 and 1a5f166
--------------------
Benchmark tpch.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━┓
┃ Query        ┃  f296e02 ┃  1a5f166 ┃       Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━┩
│ QQuery 1     │ 445.57ms │ 442.82ms │    no change │
│ QQuery 2     │  58.97ms │  59.72ms │    no change │
│ QQuery 3     │ 146.73ms │ 145.81ms │    no change │
│ QQuery 4     │  88.97ms │ 114.91ms │ 1.29x slower │
│ QQuery 5     │ 199.92ms │ 233.32ms │ 1.17x slower │
│ QQuery 6     │ 105.78ms │ 104.93ms │    no change │
│ QQuery 7     │ 283.15ms │ 280.21ms │    no change │
│ QQuery 8     │ 198.17ms │ 194.26ms │    no change │
│ QQuery 9     │ 292.63ms │ 294.16ms │    no change │
│ QQuery 10    │ 242.26ms │ 235.64ms │    no change │
│ QQuery 11    │  62.63ms │  63.30ms │    no change │
│ QQuery 12    │ 125.92ms │ 126.08ms │    no change │
│ QQuery 13    │ 183.91ms │ 187.21ms │    no change │
│ QQuery 14    │ 130.30ms │ 129.29ms │    no change │
│ QQuery 15    │ 189.46ms │ 200.74ms │ 1.06x slower │
│ QQuery 16    │  52.18ms │  52.48ms │    no change │
│ QQuery 17    │ 314.09ms │ 309.43ms │    no change │
│ QQuery 18    │ 444.25ms │ 445.22ms │    no change │
│ QQuery 19    │ 231.60ms │ 231.59ms │    no change │
│ QQuery 20    │ 192.11ms │ 193.79ms │    no change │
│ QQuery 21    │ 330.28ms │ 326.46ms │    no change │
│ QQuery 22    │  55.57ms │  54.39ms │    no change │
└──────────────┴──────────┴──────────┴──────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Benchmark Summary      ┃           ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ Total Time (f296e02)   │ 4374.45ms │
│ Total Time (1a5f166)   │ 4425.76ms │
│ Average Time (f296e02) │  198.84ms │
│ Average Time (1a5f166) │  201.17ms │
│ Queries Faster         │         0 │
│ Queries Slower         │         3 │
│ Queries with No Change │        19 │
└────────────────────────┴───────────┘