[Proposal] Visualize the query plan and query profile #5474
Description
Is your feature request related to a problem? Please describe.
Currently, Doris query plan and query profile are only be viewed in plain text format, which is not easy
to view and analyze the relationship of each node in plan or profile tree.
So I decide to visualize them to help user analyze them more easily.
Describe the solution you'd like
Query Plan
Query plan is a relative simple structure. Each plan fragment in the query plan is already a tree structure.
What we do is to connect all fragments together (DataSink -> ExchangeNode), and then draw a tree graph
for it.
┌──────────────┐
│[2: DATA SINK]│
│Fragment: 2] │
│RESULT SINK │
└──────────────┘
│
│
┌─────────────────────────────────┐
│[2: HASH JOIN] │
│Fragment: 2] │
│join op: INNER JOIN (PARTITIONED)│
└─────────────────────────────────┘
┌───────────────┴──────────────┐
│ │
┌─────────────┐ ┌─────────────┐
│[3: EXCHANGE]│ │[4: EXCHANGE]│
│Fragment: 2] │ │Fragment: 2] │
└─────────────┘ └─────────────┘
└┐ └┐
│ │
┌────────────────────────────┐ ┌────────────────────────────┐
│[3: DATA SINK] │ │[4: DATA SINK] │
│Fragment: 0] │ │Fragment: 1] │
│STREAM DATA SINK │ │STREAM DATA SINK │
│ EXCHANGE ID: 03 │ │ EXCHANGE ID: 04 │
│ HASH_PARTITIONED: `a`.`k1`│ │ HASH_PARTITIONED: `b`.`k1`│
└────────────────────────────┘ └────────────────────────────┘
┌┘ ┌┘
│ │
┌─────────────────┐ ┌─────────────────┐
│[0: OlapScanNode]│ │[1: OlapScanNode]│
│Fragment: 0] │ │Fragment: 1] │
│TABLE: tbl1 │ │TABLE: tbl1 │
└─────────────────┘ └─────────────────┘
Query Profile
Query profile is more complicate because it does not only show the relationship of each plan fragments.
But it also shows all instances in each plan fragments.
So I plan to divide the presentation of the profile into several steps:
-
Fragment Tree
The fragment tree is a tree used to display the overall query execution structure. They show the connection relationship
between each fragment, and the connection relationship of the execution nodes within each fragment.
And at the same time, the execution time of the instance that takes the longest in each Fragment will be displayed.Through this figure, the user can understand the overall execution structure, and can further expand a fragment to
continue analyzing the profile according to the maximum execution time.
┌──────────────────────┐
│[-1: DataBufferSender]│
│Fragment: 0 │
│MaxActiveTime: 7.850ms│
└──────────────────────┘
│
│
┌───────────────────┐
│[2: HASH_JOIN_NODE]│
│Fragment: 0 │
└───────────────────┘
┌────────────┴─────────────┐
│ │
┌──────────────────┐ ┌──────────────────┐
│[3: EXCHANGE_NODE]│ │[4: EXCHANGE_NODE]│
│Fragment: 0 │ │Fragment: 0 │
└──────────────────┘ └──────────────────┘
│ │
│ │
┌────────────────────────┐ ┌────────────────────────┐
│[3: DataStreamSender] │ │[4: DataStreamSender] │
│Fragment: 2 │ │Fragment: 1 │
│MaxActiveTime: 218.157us│ │MaxActiveTime: 562.931us│
└────────────────────────┘ └────────────────────────┘
┌┘ ┌┘
│ │
┌───────────────────┐ ┌───────────────────┐
│[0: OLAP_SCAN_NODE]│ │[1: OLAP_SCAN_NODE]│
│Fragment: 2 │ │Fragment: 1 │
└───────────────────┘ └───────────────────┘
│ │
│ │
┌─────────────┐ ┌─────────────┐
│[OlapScanner]│ │[OlapScanner]│
│Fragment: 2 │ │Fragment: 1 │
└─────────────┘ └─────────────┘
│ │
│ │
┌─────────────────┐ ┌─────────────────┐
│[SegmentIterator]│ │[SegmentIterator]│
│Fragment: 2 │ │Fragment: 1 │
└─────────────────┘ └─────────────────┘
-
Instance List
The user can continue to expand a specified fragment, and then get a list of all instances of this fragment.
The list contains instance id, instance execution host and instance execution time.
Further, the user can continue to expand an instance for further analysis.
+----------------------------------+------------------+------------+
| Instances | Host | ActiveTime |
+----------------------------------+------------------+------------+
| f8db008ccdf4ed8-807102c30d795eaa | 10.81.85.89:9062 | 7.850ms |
+----------------------------------+------------------+------------+
-
Instance Tree
Instance Tree will display the detailed information of each execution node in an instance.
Here is the most fine-grained information of the profile
┌──────────────────────────────┐
│[-1: DataBufferSender] │
│(Active: 0ns, non-child: 0.00)│
│ - Counters: │
│ - AppendBatchTime: 0ns │
│ - NumSentRows: 0 │
└──────────────────────────────┘
│
│
┌───────────────────────────────────┐
│[2: HASH_JOIN_NODE] │
│(Active: 7.842ms, non-child: 0.00) │
│ - Counters: │
│ - BuildBuckets: 1.024K (1024)│
│ - BuildRows: 0 │
│ - BuildTime: 6.989us │
│ - LoadFactor: 0.00 │
│ - PeakMemoryUsage: 48.00 KB │
│ - ProbeRows: 0 │
│ - ProbeTime: 0ns │
│ - PushDownComputeTime: 0ns │
│ - PushDownTime: 0ns │
│ - RowsReturned: 0 │
│ - RowsReturnedRate: 0 │
└───────────────────────────────────┘
┌─────────────────────┴──────────────────────┐
│ │
┌──────────────────────────────────────────┐ ┌──────────────────────────────────────────┐
│[3: EXCHANGE_NODE] │ │[4: EXCHANGE_NODE] │
│(Active: 7.779ms, non-child: 23.54) │ │(Active: 6.440ms, non-child: 19.49) │
│ - Counters: │ │ - Counters: │
│ - BytesReceived: 0.00 │ │ - BytesReceived: 0.00 │
│ - ConvertRowBatchTime: 572ns │ │ - ConvertRowBatchTime: 623ns │
│ - DataArrivalWaitTime: 7.771ms │ │ - DataArrivalWaitTime: 6.432ms │
│ - DeserializeRowBatchTimer: 0ns │ │ - DeserializeRowBatchTimer: 0ns │
│ - FirstBatchArrivalWaitTime: 7.771ms│ │ - FirstBatchArrivalWaitTime: 6.431ms│
│ - PeakMemoryUsage: 0.00 │ │ - PeakMemoryUsage: 0.00 │
│ - RowsReturned: 0 │ │ - RowsReturned: 0 │
│ - RowsReturnedRate: 0 │ │ - RowsReturnedRate: 0 │
│ - SendersBlockedTotalTimer(*): 0ns │ │ - SendersBlockedTotalTimer(*): 0ns │
└──────────────────────────────────────────┘ └──────────────────────────────────────────┘
Describe alternatives you've considered
First, I will only implement this in text graph.
Next, It can be show in web frontend.