[Proposal] Get the hot spot of the doris cluster.

[chaoyli]

Nowadays, We have no insight about the hotspot the cluster.
During the time, we don't know the scanner rows of one table during time.
Now we should collect the scanner rows of the table and scanner bytes of the table for query.
Using this information, we can got the insight about the hot spot of cluster.

[chaoyli]

The metric is like this

[
   {
      "tags":{
         "table_id":10010,
         "metric":"scanner_rows"
      },
      "value":1000,
      "unit":"number"
   },
   {
      "tags":{
         "table_id":10010,
         "metric":"scanner_bytes"
      },
      "value":1024,
      "unit":"bytes"
   }
]

Supposing there are 10000 tablets in one BE node and collecting metrics at every 15 seconds.
Every time we will gather 10000 * 200 Bytes = 2MB, one hundred nodes will collect 200MB, this will be consume huge bandwidth. And also user will take care of table granularity than tablet. So we will collect table metrics instead of tablet metrics.
For metric in one BE node, I will use a map to store the information.

std::unordered_map<table_id, metric> table_metrics

The incremental metric value should be calculated at every collecting interval.
Like this

2020-05-08 14:00:00  10000  1024(scanner_rows)
2020-05-08 14:00:15   10000  2048(scanner_rows)
The incremental value at this cycle is 1024.

[chaoyli]

There also exists another function using log.
If every query collecting the tablet scanner rows/bytes, and write it to log.
We can access the log to get the message, it needs log analysis which is not integrated in Doris.
So I will use the above method to solve this problem.

[imay]

This is a good job.
However I think something maybe discussed clearly before writing code.

1. Who will update this metrics?
2. How will updater get the table id?
3. How will the final metric be summary?

[chaoyli]

This is a good job.
However I think something maybe discussed clearly before writing code.

1. Who will update this metrics?
2. How will updater get the table id?
3. How will the final metric be summary?

1. Scanner Thread will update the metrics, it is the same as the metric like the query profile.
2. Nowadays, tablet meta has save the table id in the metadata, so new table will know the table_id.
3. If metric has been collected by agent, you can store to another database, or can be summed by a central node. There are ok both.

[morningman]

I think we could refer to
https://pingcap.com/blog-cn/observability-of-distributed-system/
and
https://cloud.google.com/bigtable/docs/keyvis-overview

to learn something