benchmark, TOC: add 4.0 TPC-H, TPC-C and Sysbench test reports#2661
Conversation
| --- | ||
| title: TiDB TPC-C Performance Test Report -- v4.0 vs. v3.0 | ||
| summary: Compares the TPC-C performance of TiDB 4.0 and TiDB 3.0 using BenchmarkSQL. | ||
| summary: Compare the TPC-H performance of TiDB 4.0 and TiDB 3.0 using BenchmarkSQL. |
| git clone https://github.com/pingcap/tidb-bench.git && cd tpch/queries | ||
| ``` | ||
|
|
||
| 2. Execute the query and record the executing time of the query. |
There was a problem hiding this comment.
Should be "Execute those queries and record the executing time of each query"?
|
|
||
| + Blue lines represent v3.0; | ||
| + Red lines represent v4.0 (data read only from TiKV); | ||
| + Yellow lines represent v4.0 (data read from TiKV and TiFlash automatically based on intelligent choice). |
There was a problem hiding this comment.
We use the term "smart selection" instead of "intelligent choice" in this doc:
https://pingcap.com/docs/stable/tiflash/use-tiflash/#smart-selection
There was a problem hiding this comment.
This TiFlash overview document uses Intelligent choice https://pingcap.com/docs/stable/tiflash/tiflash-overview/#intelligent-choice
|
Please help review this PR, thx~ @CaitinChen FYI @TomShawn |
|
@JaySon-Huang, @scsldb, @CaitinChen, @shuke987, PTAL. |
There was a problem hiding this comment.
My review on the TPC-H doc is done. But I think we need to add numbers to show the improvements.
And, just like other benchmark reports, we should point put "TiDB version: v3.0.13 vs. v4.0.0."
@JaySon-Huang PTAL
| title: TiDB TPC-H Performance Test Report -- v4.0 vs. v3.0 | ||
| summary: Compare the TPC-H performance of TiDB 4.0 and TiDB 3.0. | ||
| category: benchmark | ||
| draft: true |
|
|
||
| ## Test purpose | ||
|
|
||
| This test aims to compare the TPC-H performances of TiDB 4.0 and TiDB 3.0 in the OLAP (Online Analytical Processing) scenario. |
There was a problem hiding this comment.
| This test aims to compare the TPC-H performances of TiDB 4.0 and TiDB 3.0 in the OLAP (Online Analytical Processing) scenario. | |
| This test aims to compare the TPC-H performances of TiDB 4.0 and TiDB 3.0 in the online analytical processing (OLAP) scenario. |
|
|
||
| This test aims to compare the TPC-H performances of TiDB 4.0 and TiDB 3.0 in the OLAP (Online Analytical Processing) scenario. | ||
|
|
||
| Because [TiFlash](/tiflash/tiflash-overview.md) is introduced in TiDB v4.0 that enhances the HTAP (Hybrid Transactional and Analytical Processing) performance of TiDB, the following performances are tested and compared in this report: |
There was a problem hiding this comment.
| Because [TiFlash](/tiflash/tiflash-overview.md) is introduced in TiDB v4.0 that enhances the HTAP (Hybrid Transactional and Analytical Processing) performance of TiDB, the following performances are tested and compared in this report: | |
| Because [TiFlash](/tiflash/tiflash-overview.md) is introduced in TiDB v4.0, which enhances TiDB's Hybrid Transactional and Analytical Processing (HTAP) capabilities, test objects in this report are as follows: |
There was a problem hiding this comment.
|
|
||
| #### Hardware prerequisite | ||
|
|
||
| To avoid TiKV and TiFlash racing for disk and I/O resources, mount the two NVMe SSD disks configured on EC2 as `/data1` and `/data2`. Deploy TiKV on `/data1` and deploy TiFlash on `/data2`. |
There was a problem hiding this comment.
| To avoid TiKV and TiFlash racing for disk and I/O resources, mount the two NVMe SSD disks configured on EC2 as `/data1` and `/data2`. Deploy TiKV on `/data1` and deploy TiFlash on `/data2`. | |
| To avoid TiKV and TiFlash racing for disk and I/O resources, mount the two NVMe SSD disks configured on EC2 to `/data1` and `/data2`. Deploy TiKV on `/data1` and deploy TiFlash on `/data2`. |
There was a problem hiding this comment.
Maybe "mount ... at "?
http://linuxcommand.org/lc3_man_pages/mount8.html
This tells the kernel to attach the filesystem found on device (which is of type type) at the directory dir.
|
|
||
| | Query ID | v3.0 | v4.0 TiKV Only | v4.0 TiKV/TiFlash Automatically | | ||
| | :-------- | :----------- | :------------ | :-------------- | | ||
| | 1 | 7.78s | 7.45s | 2.09s | |
There was a problem hiding this comment.
| | 1 | 7.78s | 7.45s | 2.09s | | |
| | 1 | 7.78 s | 7.45 s | 2.09 s | |
There was a problem hiding this comment.
Same comments for the following lines.
| | 21 | 11.12s | 6.23s | 6.26s | | ||
| | 22 | 4.49s | 3.05s | 2.31s | | ||
|
|
||
|  |
There was a problem hiding this comment.
|  | |
|  |
There was a problem hiding this comment.
"_" is not allowed in image names.
| + Blue lines represent v3.0; | ||
| + Red lines represent v4.0 (data read only from TiKV); | ||
| + Yellow lines represent v4.0 (data read from TiKV and TiFlash automatically based on intelligent choice). | ||
| + The ordinate represents the processing time of the query. The lower the ordinate, the better the performance. |
There was a problem hiding this comment.
| + The ordinate represents the processing time of the query. The lower the ordinate, the better the performance. | |
| + The y-axis represents the execution time of the query. The less the time, the better the performance. |
|
|
||
| Result description: | ||
|
|
||
| + **v4.0 TiKV Only** means that TiDB reads data only from TiKV. Comparing this result with that of v3.0, you can learn the TPC-H performance increase after TiDB and TiKV are upgraded to v4.0. |
There was a problem hiding this comment.
| + **v4.0 TiKV Only** means that TiDB reads data only from TiKV. Comparing this result with that of v3.0, you can learn the TPC-H performance increase after TiDB and TiKV are upgraded to v4.0. | |
| + **v4.0 TiKV Only** means that TiDB reads data only from TiKV. The result shows that the TPC-H performance increased after TiDB and TiKV are upgraded to v4.0. |
| Result description: | ||
|
|
||
| + **v4.0 TiKV Only** means that TiDB reads data only from TiKV. Comparing this result with that of v3.0, you can learn the TPC-H performance increase after TiDB and TiKV are upgraded to v4.0. | ||
| + **v4.0 TiKV/TiFlash Automatically** means that the TiDB optimizer automatically determines whether to use the TiFlash replica according to the cost estimation. Comparing this result with that of v3.0, you can learn the TPC-H performance increase in the full HTAP form of v4.0. |
There was a problem hiding this comment.
| + **v4.0 TiKV/TiFlash Automatically** means that the TiDB optimizer automatically determines whether to use the TiFlash replica according to the cost estimation. Comparing this result with that of v3.0, you can learn the TPC-H performance increase in the full HTAP form of v4.0. | |
| + **v4.0 TiKV/TiFlash Automatically** means that the TiDB optimizer automatically determines whether to read data from the TiFlash replica according to the cost estimation. The result shows that the TPC-H performance increased in the full HTAP form of v4.0. |
CaitinChen
left a comment
There was a problem hiding this comment.
My review on the TPC-C doc is done.
| ## Test purpose | ||
|
|
||
| This test aims to compare the TPC-C performance of TiDB 4.0 and TiDB 3.0. | ||
| This test aims to compare the TPC-C performances of TiDB 4.0 and TiDB 3.0 in the OLTP (Online Transactional Processing) scenario. |
There was a problem hiding this comment.
| This test aims to compare the TPC-C performances of TiDB 4.0 and TiDB 3.0 in the OLTP (Online Transactional Processing) scenario. | |
| This test aims to compare the TPC-C performances of TiDB 4.0 and TiDB 3.0 in the Online Transactional Processing (OLTP) scenario. |
| git clone https://github.com/pingcap/benchmarksql && cd benchmarksql && ant | ||
| ``` | ||
|
|
||
| 2. Enter the `run` directory, edit the `props.mysql` file according to the actual situation. Modify the `conn`, `warehouses`, `loadWorkers`, `terminals`, and `runMins` configuration items. |
There was a problem hiding this comment.
| 2. Enter the `run` directory, edit the `props.mysql` file according to the actual situation. Modify the `conn`, `warehouses`, `loadWorkers`, `terminals`, and `runMins` configuration items. | |
| 2. Enter the `run` directory, edit the `props.mysql` file according to the actual situation, and modify the `conn`, `warehouses`, `loadWorkers`, `terminals`, and `runMins` configuration items. |
|
|
||
| 4. Execute the `runSQL.sh ./props.mysql sql.mysql/indexCreates.sql` command. | ||
|
|
||
| 5. Execute MySQL client and execute the `analyze table` statement on every table. |
There was a problem hiding this comment.
| 5. Execute MySQL client and execute the `analyze table` statement on every table. | |
| 5. Run the MySQL client and execute the `analyze table` statement on every table. |
| According to the test statistics, the TPC-C performance of TiDB v4.0 has **increased by 50%** compared with that of TiDB v3.0. | ||
|
|
||
| According to the testing statistics, the performance of TiDB 4.0.0-rc **has increased by 45%** than that of TiDB 3.0.8. | ||
|  |
There was a problem hiding this comment.
|  | |
|  |
There was a problem hiding this comment.
"_" is not allowed in our figure names.
CaitinChen
left a comment
There was a problem hiding this comment.
My review on the Sysbench doc is done.
|
|
||
| ## Test purpose | ||
|
|
||
| This test aims to compare the Sysbench performances of TiDB 4.0 and TiDB 3.0 in the OLTP (Online Transactional Processing) scenario. |
There was a problem hiding this comment.
| This test aims to compare the Sysbench performances of TiDB 4.0 and TiDB 3.0 in the OLTP (Online Transactional Processing) scenario. | |
| This test aims to compare the Sysbench performances of TiDB 4.0 and TiDB 3.0 in the Online Transactional Processing (OLTP) scenario. |
|
|
||
| | Service type | Software version | | ||
| |:----------|:-----------| | ||
| | PD | 3.0、4.0 | |
There was a problem hiding this comment.
| | PD | 3.0、4.0 | | |
| | PD | 3.0 and 4.0 | |
| | Service type | Software version | | ||
| |:----------|:-----------| | ||
| | PD | 3.0、4.0 | | ||
| | TiDB | 3.0、4.0 | |
There was a problem hiding this comment.
| | TiDB | 3.0、4.0 | | |
| | TiDB | 3.0 and 4.0 | |
| |:----------|:-----------| | ||
| | PD | 3.0、4.0 | | ||
| | TiDB | 3.0、4.0 | | ||
| | TiKV | 3.0、4.0 | |
There was a problem hiding this comment.
| | TiKV | 3.0、4.0 | | |
| | TiKV | 3.0 and 4.0 | |
| {{< copyable "" >}} | ||
|
|
||
| ```yaml | ||
| log.level: “error” |
There was a problem hiding this comment.
| log.level: “error” | |
| log.level: "error" |
|
|
||
| Compared with v3.0, the Point Select performance of TiDB v4.0 has increased by 14%. | ||
|
|
||
|  |
There was a problem hiding this comment.
|  | |
|  |
There was a problem hiding this comment.
"_" is not allowed in our figure names.
|
|
||
| Compared with v3.0, the Update Non-index performance of TiDB v4.0 has increased by 15%. | ||
|
|
||
|  |
There was a problem hiding this comment.
|  | |
|  |
|
|
||
| Compared with v3.0, the Update Index performance of TiDB v4.0 has increased by 17%. | ||
|
|
||
|  |
There was a problem hiding this comment.
|  | |
|  |
|
|
||
|  | ||
|
|
||
| ### Read Write performance |
There was a problem hiding this comment.
| ### Read Write performance | |
| ### Read-write performance |
|
|
||
| ### Read Write performance | ||
|
|
||
| Compared with v3.0, the Read Write performance of TiDB v4.0 has increased by 31%. |
There was a problem hiding this comment.
| Compared with v3.0, the Read Write performance of TiDB v4.0 has increased by 31%. | |
| Compared with v3.0, the read-write performance of TiDB v4.0 has increased by 31%. |
|
@CaitinChen @JaySon-Huang Comments addressed, PTAL again. |
| 4. Extract the tpmC data of New Order from the result. | ||
|
|
||
| ## Test result | ||
| ### Test result |
There was a problem hiding this comment.
| ### Test result | |
| ## Test result |
| ``` | ||
|
|
||
| ### Cluster topology | ||
| ### Test plan |
There was a problem hiding this comment.
| ### Test plan | |
| ## Test plan |
| set global tidb_index_lookup_join_concurrency = 16; | ||
| ``` | ||
|
|
||
| ### Test plan |
There was a problem hiding this comment.
| ### Test plan | |
| ## Test plan |
|
|
||
| ### Test plan | ||
|
|
||
| #### Hardware prerequisite |
There was a problem hiding this comment.
| #### Hardware prerequisite | |
| ### Hardware prerequisite |
|
|
||
| To avoid TiKV and TiFlash racing for disk and I/O resources, mount the two NVMe SSD disks configured on EC2 to `/data1` and `/data2`. Deploy TiKV on `/data1` and deploy TiFlash on `/data2`. | ||
|
|
||
| #### Test process |
There was a problem hiding this comment.
| #### Test process | |
| ### Test process |
|
|
||
| 4. Extract and organize the data of query execution time. | ||
|
|
||
| ### Test result |
There was a problem hiding this comment.
| ### Test result | |
| ## Test result |
Signed-off-by: sre-bot <sre-bot@pingcap.com>
|
cherry pick to release-4.0 in PR #2711 |
What is changed, added or deleted? (Required)
add 4.0 TPC-H, TPC-C and Sysbench test reports.
Which TiDB version(s) do your changes apply to? (Required)
If you select two or more versions from above, to trigger the bot to cherry-pick this PR to your desired release version branch(es), you must add corresponding labels such as needs-cherry-pick-4.0, needs-cherry-pick-3.1, needs-cherry-pick-3.0, and needs-cherry-pick-2.1.
What is the related PR or file link(s)?