Skip to content

[fix](profile) modify load profile some bugs and docs #17533

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
zhannngchen merged 2 commits into from
Mar 9, 2023
Merged

[fix](profile) modify load profile some bugs and docs #17533

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
4a63c62
(profile) modify load profile some bug and docs
nextdreamblue Mar 7, 2023
c389719
fix ut
nextdreamblue Mar 8, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
75 changes: 53 additions & 22 deletions docs/en/docs/advanced/best-practice/import-analysis.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -68,12 +68,43 @@ Then submit a Broker Load import request and wait until the import execution com
We can get the Profile list first with the following command:

```sql
mysql> show load profile "/";
+---------+------+-----------+------+------------+- --------------------+---------------------------------+------- ----+------------+
| QueryId | User | DefaultDb | SQL | QueryType | StartTime | EndTime | TotalTime | QueryState |
+---------+------+-----------+------+------------+- --------------------+---------------------------------+------- ----+------------+
| 10441 | N/A | N/A | N/A | Load | 2021-04-10 22:15:37 | 2021-04-10 22:18:54 | 3m17s | N/A |
+---------+------+-----------+------+------------+- --------------------+---------------------------------+------- ----+------------+
mysql> show load profile "/"\G
*************************** 1. row ***************************
JobId: 20010
QueryId: 980014623046410a-af5d36f23381017f
User: root
DefaultDb: default_cluster:test
SQL: LOAD LABEL xxx
QueryType: Load
StartTime: 2023-03-07 19:48:24
EndTime: 2023-03-07 19:50:45
TotalTime: 2m21s
QueryState: N/A
TraceId:
AnalysisTime: NULL
PlanTime: NULL
ScheduleTime: NULL
FetchResultTime: NULL
WriteResultTime: NULL
WaitAndFetchResultTime: NULL
*************************** 2. row ***************************
JobId: N/A
QueryId: 7cc2d0282a7a4391-8dd75030185134d8
User: root
DefaultDb: default_cluster:test
SQL: insert into xxx
QueryType: Load
StartTime: 2023-03-07 19:49:15
EndTime: 2023-03-07 19:49:15
TotalTime: 102ms
QueryState: OK
TraceId:
AnalysisTime: 825.277us
PlanTime: 4.126ms
ScheduleTime: N/A
FetchResultTime: 0ns
WriteResultTime: 0ns
WaitAndFetchResultTime: N/A
````

This command will list all currently saved import profiles. Each line corresponds to one import. where the QueryId column is the ID of the import job. This ID can also be viewed through the SHOW LOAD statement. We can select the QueryId corresponding to the Profile we want to see to see the specific situation.
Expand All @@ -87,15 +118,15 @@ This command will list all currently saved import profiles. Each line correspond


```sql
mysql> show load profile "/10441";
mysql> show load profile "/980014623046410a-af5d36f23381017f";
+-----------------------------------+------------+
| TaskId | ActiveTime |
| TaskId | ActiveTime |
+-----------------------------------+------------+
| 980014623046410a-88e260f0c43031f1 | 3m14s |
| 980014623046410a-af5d36f23381017f | 3m14s |
+-----------------------------------+------------+
````

As shown in the figure above, it means that the import job 10441 has a total of one subtask, in which ActiveTime indicates the execution time of the longest instance in this subtask.
As shown in the figure above, it means that the import job `980014623046410a-af5d36f23381017f` has a total of one subtask, in which ActiveTime indicates the execution time of the longest instance in this subtask.

2. View the Instance overview of the specified subtask

Expand All @@ -104,18 +135,18 @@ As shown in the figure above, it means that the import job 10441 has a total of


```sql
mysql> show load profile "/10441/980014623046410a-88e260f0c43031f1";
+-----------------------------------+------------- -----+------------+
| Instances | Host | ActiveTime |
+-----------------------------------+------------- -----+------------+
| 980014623046410a-88e260f0c43031f2 | 10.81.85.89:9067 | 3m7s |
| 980014623046410a-88e260f0c43031f3 | 10.81.85.89:9067 | 3m6s |
| 980014623046410a-88e260f0c43031f4 | 10.81.85.89:9067 | 3m10s |
| 980014623046410a-88e260f0c43031f5 | 10.81.85.89:9067 | 3m14s |
+-----------------------------------+------------- -----+------------+
mysql> show load profile "/980014623046410a-af5d36f23381017f/980014623046410a-af5d36f23381017f";
+-----------------------------------+------------------+------------+
| Instances | Host | ActiveTime |
+-----------------------------------+------------------+------------+
| 980014623046410a-88e260f0c43031f2 | 10.81.85.89:9067 | 3m7s |
| 980014623046410a-88e260f0c43031f3 | 10.81.85.89:9067 | 3m6s |
| 980014623046410a-88e260f0c43031f4 | 10.81.85.89:9067 | 3m10s |
| 980014623046410a-88e260f0c43031f5 | 10.81.85.89:9067 | 3m14s |
+-----------------------------------+------------------+------------+
````

This shows the time-consuming of four instances of the subtask 980014623046410a-88e260f0c43031f1, and also shows the execution node where the instance is located.
This shows the time-consuming of four instances of the subtask 980014623046410a-af5d36f23381017f, and also shows the execution node where the instance is located.

3. View the specific Instance

Expand All @@ -124,7 +155,7 @@ This shows the time-consuming of four instances of the subtask 980014623046410a-


```sql
mysql> show load profile "/10441/980014623046410a-88e260f0c43031f1/980014623046410a-88e260f0c43031f5"\G
mysql> show load profile "/980014623046410a-af5d36f23381017f/980014623046410a-af5d36f23381017f/980014623046410a-88e260f0c43031f5"\G
**************************** 1. row ******************** ******
Instance:
┌-----------------------------------------┐
Expand Down Expand Up @@ -166,6 +197,6 @@ This shows the time-consuming of four instances of the subtask 980014623046410a-
└------------------------------------------------- ----┘
````

The figure above shows the specific profiles of each operator of Instance 980014623046410a-88e260f0c43031f5 in subtask 980014623046410a-88e260f0c43031f1.
The figure above shows the specific profiles of each operator of Instance 980014623046410a-af5d36f23381017f in subtask 980014623046410a-88e260f0c43031f5.

Through the above three steps, we can gradually check the execution bottleneck of an import task.
50 changes: 40 additions & 10 deletions docs/en/docs/sql-manual/sql-reference/Show-Statements/SHOW-LOAD-PROFILE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,30 +59,60 @@ This command will list all currently saved import profiles. Each line correspond
1. List all Load Profiles

```sql
mysql> show load profile "/";
+---------+------+-----------+------+-----------+---------------------+---------------------+-----------+------------+
| QueryId | User | DefaultDb | SQL | QueryType | StartTime | EndTime | TotalTime | QueryState |
+---------+------+-----------+------+-----------+---------------------+---------------------+-----------+------------+
| 10441 | N/A | N/A | N/A | Load | 2021-04-10 22:15:37 | 2021-04-10 22:18:54 | 3m17s | N/A |
+---------+------+-----------+------+-----------+---------------------+---------------------+-----------+------------+
2 rows in set (0.00 sec)
mysql> show load profile "/"\G
*************************** 1. row ***************************
JobId: 20010
QueryId: 980014623046410a-af5d36f23381017f
User: root
DefaultDb: default_cluster:test
SQL: LOAD LABEL xxx
QueryType: Load
StartTime: 2023-03-07 19:48:24
EndTime: 2023-03-07 19:50:45
TotalTime: 2m21s
QueryState: N/A
TraceId:
AnalysisTime: NULL
PlanTime: NULL
ScheduleTime: NULL
FetchResultTime: NULL
WriteResultTime: NULL
WaitAndFetchResultTime: NULL
*************************** 2. row ***************************
JobId: N/A
QueryId: 7cc2d0282a7a4391-8dd75030185134d8
User: root
DefaultDb: default_cluster:test
SQL: insert into xxx
QueryType: Load
StartTime: 2023-03-07 19:49:15
EndTime: 2023-03-07 19:49:15
TotalTime: 102ms
QueryState: OK
TraceId:
AnalysisTime: 825.277us
PlanTime: 4.126ms
ScheduleTime: N/A
FetchResultTime: 0ns
WriteResultTime: 0ns
WaitAndFetchResultTime: N/A
````

2. View an overview of the subtasks with imported jobs:

```sql
mysql> show load profile "/10441";
mysql> show load profile "/980014623046410a-af5d36f23381017f";
+-----------------------------------+------------+
| TaskId | ActiveTime |
+-----------------------------------+------------+
| 980014623046410a-88e260f0c43031f1 | 3m14s |
| 980014623046410a-af5d36f23381017f | 3m14s |
+-----------------------------------+------------+
````

3. View the Instance overview of the specified subtask

```sql
mysql> show load profile "/10441/980014623046410a-88e260f0c43031f1";
mysql> show load profile "/980014623046410a-af5d36f23381017f/980014623046410a-af5d36f23381017f/980014623046410a-88e260f0c43031f5"\G
+-----------------------------------+------------------+------------+
| Instances | Host | ActiveTime |
+-----------------------------------+------------------+------------+
Expand Down
57 changes: 44 additions & 13 deletions docs/zh-CN/docs/advanced/best-practice/import-analysis.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,12 +67,43 @@ SET is_report_success=true;
我们可以通过如下命令先获取 Profile 列表：

```sql
mysql> show load profile "/";
+---------+------+-----------+------+-----------+---------------------+---------------------+-----------+------------+
| QueryId | User | DefaultDb | SQL | QueryType | StartTime | EndTime | TotalTime | QueryState |
+---------+------+-----------+------+-----------+---------------------+---------------------+-----------+------------+
| 10441 | N/A | N/A | N/A | Load | 2021-04-10 22:15:37 | 2021-04-10 22:18:54 | 3m17s | N/A |
+---------+------+-----------+------+-----------+---------------------+---------------------+-----------+------------+
mysql> show load profile "/"\G
*************************** 1. row ***************************
JobId: 20010
QueryId: 980014623046410a-af5d36f23381017f
User: root
DefaultDb: default_cluster:test
SQL: LOAD LABEL xxx
QueryType: Load
StartTime: 2023-03-07 19:48:24
EndTime: 2023-03-07 19:50:45
TotalTime: 2m21s
QueryState: N/A
TraceId:
AnalysisTime: NULL
PlanTime: NULL
ScheduleTime: NULL
FetchResultTime: NULL
WriteResultTime: NULL
WaitAndFetchResultTime: NULL
*************************** 2. row ***************************
JobId: N/A
QueryId: 7cc2d0282a7a4391-8dd75030185134d8
User: root
DefaultDb: default_cluster:test
SQL: insert into xxx
QueryType: Load
StartTime: 2023-03-07 19:49:15
EndTime: 2023-03-07 19:49:15
TotalTime: 102ms
QueryState: OK
TraceId:
AnalysisTime: 825.277us
PlanTime: 4.126ms
ScheduleTime: N/A
FetchResultTime: 0ns
WriteResultTime: 0ns
WaitAndFetchResultTime: N/A
```

这个命令会列出当前保存的所有导入 Profile。每行对应一个导入。其中 QueryId 列为导入作业的 ID。这个 ID 也可以通过 SHOW LOAD 语句查看拿到。我们可以选择我们想看的 Profile 对应的 QueryId，查看具体情况。
Expand All @@ -86,15 +117,15 @@ mysql> show load profile "/";


```sql
mysql> show load profile "/10441";
mysql> show load profile "/980014623046410a-af5d36f23381017f";
+-----------------------------------+------------+
| TaskId | ActiveTime |
+-----------------------------------+------------+
| 980014623046410a-88e260f0c43031f1 | 3m14s |
| 980014623046410a-af5d36f23381017f | 3m14s |
+-----------------------------------+------------+
```

如上图，表示 10441 这个导入作业总共有一个子任务，其中 ActiveTime 表示这个子任务中耗时最长的 Instance 的执行时间。
如上图，表示 `980014623046410a-af5d36f23381017f` 这个导入作业总共有一个子任务，其中 ActiveTime 表示这个子任务中耗时最长的 Instance 的执行时间。

2. 查看指定子任务的 Instance 概况

Expand All @@ -103,7 +134,7 @@ mysql> show load profile "/";


```sql
mysql> show load profile "/10441/980014623046410a-88e260f0c43031f1";
mysql> show load profile "/980014623046410a-af5d36f23381017f/980014623046410a-af5d36f23381017f";
+-----------------------------------+------------------+------------+
| Instances | Host | ActiveTime |
+-----------------------------------+------------------+------------+
Expand All @@ -114,7 +145,7 @@ mysql> show load profile "/";
+-----------------------------------+------------------+------------+
```

这里展示了 980014623046410a-88e260f0c43031f1 这个子任务的四个 Instance 耗时，并且还展示了 Instance 所在的执行节点。
这里展示了 980014623046410a-af5d36f23381017f 这个子任务的四个 Instance 耗时，并且还展示了 Instance 所在的执行节点。

3. 查看具体 Instance

Expand All @@ -123,7 +154,7 @@ mysql> show load profile "/";


```sql
mysql> show load profile "/10441/980014623046410a-88e260f0c43031f1/980014623046410a-88e260f0c43031f5"\G
mysql> show load profile "/980014623046410a-af5d36f23381017f/980014623046410a-af5d36f23381017f/980014623046410a-88e260f0c43031f5"\G
*************************** 1. row ***************************
Instance:
┌-----------------------------------------┐
Expand Down Expand Up @@ -165,6 +196,6 @@ mysql> show load profile "/";
└-----------------------------------------------------┘
```

上图展示了子任务 980014623046410a-88e260f0c43031f1 中，Instance 980014623046410a-88e260f0c43031f5 的各个算子的具体 Profile。
上图展示了子任务 980014623046410a-af5d36f23381017f 中，Instance 980014623046410a-88e260f0c43031f5 的各个算子的具体 Profile。

通过以上3个步骤，我们可以逐步排查一个导入任务的执行瓶颈。
52 changes: 41 additions & 11 deletions docs/zh-CN/docs/sql-manual/sql-reference/Show-Statements/SHOW-LOAD-PROFILE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,30 +59,60 @@ show load profile "/[queryId]"
1. 列出所有的 Load Profile

```sql
mysql> show load profile "/";
+---------+------+-----------+------+-----------+---------------------+---------------------+-----------+------------+
| QueryId | User | DefaultDb | SQL | QueryType | StartTime | EndTime | TotalTime | QueryState |
+---------+------+-----------+------+-----------+---------------------+---------------------+-----------+------------+
| 10441 | N/A | N/A | N/A | Load | 2021-04-10 22:15:37 | 2021-04-10 22:18:54 | 3m17s | N/A |
+---------+------+-----------+------+-----------+---------------------+---------------------+-----------+------------+
2 rows in set (0.00 sec)
mysql> show load profile "/"\G
*************************** 1. row ***************************
JobId: 20010
QueryId: 980014623046410a-af5d36f23381017f
User: root
DefaultDb: default_cluster:test
SQL: LOAD LABEL xxx
QueryType: Load
StartTime: 2023-03-07 19:48:24
EndTime: 2023-03-07 19:50:45
TotalTime: 2m21s
QueryState: N/A
TraceId:
AnalysisTime: NULL
PlanTime: NULL
ScheduleTime: NULL
FetchResultTime: NULL
WriteResultTime: NULL
WaitAndFetchResultTime: NULL
*************************** 2. row ***************************
JobId: N/A
QueryId: 7cc2d0282a7a4391-8dd75030185134d8
User: root
DefaultDb: default_cluster:test
SQL: insert into xxx
QueryType: Load
StartTime: 2023-03-07 19:49:15
EndTime: 2023-03-07 19:49:15
TotalTime: 102ms
QueryState: OK
TraceId:
AnalysisTime: 825.277us
PlanTime: 4.126ms
ScheduleTime: N/A
FetchResultTime: 0ns
WriteResultTime: 0ns
WaitAndFetchResultTime: N/A
```

2. 查看有导入作业的子任务概况：

```sql
mysql> show load profile "/10441";
mysql> show load profile "/980014623046410a-af5d36f23381017f";
+-----------------------------------+------------+
| TaskId | ActiveTime |
+-----------------------------------+------------+
| 980014623046410a-88e260f0c43031f1 | 3m14s |
| 980014623046410a-af5d36f23381017f | 3m14s |
+-----------------------------------+------------+
```

3. 查看指定子任务的 Instance 概况

```sql
mysql> show load profile "/10441/980014623046410a-88e260f0c43031f1";
mysql> show load profile "/980014623046410a-af5d36f23381017f/980014623046410a-af5d36f23381017f";
+-----------------------------------+------------------+------------+
| Instances | Host | ActiveTime |
+-----------------------------------+------------------+------------+
Expand All @@ -96,7 +126,7 @@ show load profile "/[queryId]"
4. 继续查看某一个具体的 Instance 上各个算子的详细 Profile

```sql
mysql> show load profile "/10441/980014623046410a-88e260f0c43031f1/980014623046410a-88e260f0c43031f5"\G
mysql> show load profile "/980014623046410a-af5d36f23381017f/980014623046410a-af5d36f23381017f/980014623046410a-88e260f0c43031f5"\G

*************************** 1. row ***************************

Expand Down
Loading