[feature wanted]现在Doris的union all好像是会集中到一台机器上进行计算，导致效率不高

[li-ang-666]

Doris版本：0.12.21

我有 N 个小SQL，每个SQL执行0.6秒左右

然后这N个SQL ，union all到一起，做一个group by，随着N的增大，从profile上来看，unionn_node的某个exchange_node的执行时间会越来越长

当N为20时，执行时间是24秒，最慢的这个exchange_node的activeTime与FirstBatchArrivalWaitTime均为23秒

通过询问dev，得出的结论可能是cpu切不过来了

是不是可以优化一下union all的代码逻辑，使其不要一次性发往一个节点？

还有一个想法，就是：我们集群目前是16核的CPU，是不是可以通过增加核心数，来提升当前union all逻辑下的性能？

[xinghuayu007]

能给一个复现的例子吗

[li-ang-666]

能给一个复现的例子吗

我给超勇哥提供了例子～

[li-ang-666]

能给一个复现的例子吗

@chaoyli 勇哥

[morningman]

There are two main problems here.

1. The Plan process is inefficient.

In the Plan phase, when multiple UNION combinations are encountered, multiple rounds of iteration will increase the plan time. In my test, it may cost 2-3 seconds to analyze and plan a SQL with 30 UNIONs.

2. Too many Fragments

A SQL may generate hundreds or thousands of Fragments, and all Fragments are currently sent to the corresponding BE separately. The common fields of these Fragments will undergo multiple serialization and deserialization operations, which results in a very high RPC processing time.
In my test, a SQL with 30 UNIONs generates 300+ fragments, and total size of these fragments is over 100MB, mainly
occupied by DescriptorTable.

Optimization Strategy:

1. For problem 1, it is necessary to analyze the corresponding code hot spots through the Java Profiler tool to optimize the analysis process. (TODO)
2. For problem 2, multiple fragments sent to the same BE can be merged and sent in batch, reducing the number of RPCs and reducing the amount of data sent. (WIP)