Skip to content

branch-3.0: [fix](nereids) topN filter: use ObjectId as map key instead of Topn node #46551 #46584

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morrySnow merged 1 commit into from
Jan 8, 2025
Merged

branch-3.0: [fix](nereids) topN filter: use ObjectId as map key instead of Topn node #46551 #46584

morrySnow merged 1 commit into from
Jan 8, 2025

Conversation

@github-actions
Copy link
Contributor

@github-actions github-actions bot commented Jan 8, 2025

Cherry-picked from #46551

@englefly
[fix](nereids) topN filter: use ObjectId as map key instead of Topn n…
b37b1d5 
…ode (#46551)

### What problem does this PR solve?
Plan node is not good to be hash map key, because two plan nodes in
different tree level may be regarded as "equal". for example, in
following tree, topn1.equals(topn2) may be true.
 Topn filter generator should distinguish them, and hence topn
node is not suitable to be used as hash map key.

topn1
    -->some node
        -->topn2
             -->other node

Related PR: #31485
@Thearas
Copy link
Contributor

Thearas commented Jan 8, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring dataroaring closed this Jan 8, 2025
@dataroaring dataroaring reopened this Jan 8, 2025
@Thearas
Copy link
Contributor

Thearas commented Jan 8, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Jan 8, 2025

TPC-H: Total hot run time: 40815 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b37b1d51028fb741350af9b121065ba55601c792, data reload: false

------ Round 1 ----------------------------------
q1	17561	7333	7231	7231
q2	2063	185	176	176
q3	10569	1101	1155	1101
q4	10560	747	704	704
q5	7731	2852	2824	2824
q6	235	146	141	141
q7	959	615	601	601
q8	9351	1903	2028	1903
q9	6576	6454	6395	6395
q10	7014	2242	2290	2242
q11	462	261	260	260
q12	416	214	205	205
q13	17765	2981	2968	2968
q14	235	211	213	211
q15	565	517	514	514
q16	718	604	626	604
q17	966	595	631	595
q18	7443	6639	6700	6639
q19	1395	1096	1104	1096
q20	470	205	203	203
q21	4054	3407	3179	3179
q22	1127	1026	1023	1023
Total cold run time: 108235 ms
Total hot run time: 40815 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7444	7227	7230	7227
q2	333	235	230	230
q3	2881	2962	2955	2955
q4	2043	1802	1784	1784
q5	5719	5753	5690	5690
q6	224	140	138	138
q7	2291	1828	1827	1827
q8	3347	3547	3517	3517
q9	8839	8848	8808	8808
q10	3566	3505	3526	3505
q11	588	507	494	494
q12	813	617	624	617
q13	9398	3138	3150	3138
q14	309	269	273	269
q15	581	534	520	520
q16	703	668	681	668
q17	1841	1625	1634	1625
q18	8223	7770	7599	7599
q19	1642	1507	1603	1507
q20	2110	1871	1837	1837
q21	5571	5339	5329	5329
q22	1194	1054	1036	1036
Total cold run time: 69660 ms
Total hot run time: 60320 ms

@doris-robot
Copy link

doris-robot commented Jan 8, 2025

TPC-DS: Total hot run time: 198103 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b37b1d51028fb741350af9b121065ba55601c792, data reload: false

query1	1314	914	909	909
query2	6234	2153	2036	2036
query3	10918	4315	4115	4115
query4	66407	29032	23911	23911
query5	5053	449	442	442
query6	409	174	179	174
query7	5637	316	319	316
query8	313	225	229	225
query9	9299	2683	2684	2683
query10	464	280	250	250
query11	17309	15213	15843	15213
query12	158	110	104	104
query13	1499	430	419	419
query14	10709	6830	7101	6830
query15	214	180	174	174
query16	7235	512	463	463
query17	1126	597	610	597
query18	1879	337	326	326
query19	226	170	172	170
query20	116	111	118	111
query21	209	108	118	108
query22	4980	4834	4764	4764
query23	34850	34292	34265	34265
query24	6160	2952	2895	2895
query25	532	427	431	427
query26	653	172	177	172
query27	1878	352	367	352
query28	4071	2480	2447	2447
query29	716	484	465	465
query30	246	166	162	162
query31	1028	816	848	816
query32	65	55	58	55
query33	397	306	276	276
query34	927	506	534	506
query35	842	753	725	725
query36	1066	976	965	965
query37	117	79	72	72
query38	4120	4004	4019	4004
query39	1610	1508	1501	1501
query40	209	98	99	98
query41	55	52	50	50
query42	114	103	102	102
query43	539	506	508	506
query44	1206	840	861	840
query45	181	167	175	167
query46	1153	732	728	728
query47	2048	1944	1959	1944
query48	492	394	382	382
query49	722	397	385	385
query50	845	437	414	414
query51	7511	7230	7124	7124
query52	103	87	87	87
query53	250	184	186	184
query54	551	444	441	441
query55	77	75	76	75
query56	274	245	236	236
query57	1255	1112	1122	1112
query58	208	201	205	201
query59	3211	3119	2963	2963
query60	286	263	243	243
query61	110	107	136	107
query62	778	695	676	676
query63	215	194	201	194
query64	1365	677	637	637
query65	3267	3204	3196	3196
query66	716	291	301	291
query67	15893	15729	15640	15640
query68	3995	576	579	576
query69	428	265	266	265
query70	1197	1098	1117	1098
query71	350	268	265	265
query72	6408	4290	3941	3941
query73	741	345	351	345
query74	10306	8965	9004	8965
query75	3325	2637	2686	2637
query76	1997	1119	1090	1090
query77	539	266	279	266
query78	10604	9835	9712	9712
query79	1392	603	596	596
query80	882	440	431	431
query81	520	243	242	242
query82	1295	111	117	111
query83	252	141	135	135
query84	290	82	76	76
query85	898	296	290	290
query86	334	309	299	299
query87	4496	4252	4341	4252
query88	3932	2411	2334	2334
query89	420	284	287	284
query90	2012	190	187	187
query91	177	145	148	145
query92	62	50	51	50
query93	1950	548	546	546
query94	788	272	283	272
query95	353	256	256	256
query96	615	280	281	280
query97	3418	3199	3255	3199
query98	215	209	196	196
query99	1594	1294	1308	1294
Total cold run time: 320405 ms
Total hot run time: 198103 ms

@doris-robot
Copy link

doris-robot commented Jan 8, 2025

ClickBench: Total hot run time: 32.46 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b37b1d51028fb741350af9b121065ba55601c792, data reload: false

query1	0.03	0.04	0.03
query2	0.07	0.02	0.04
query3	0.23	0.06	0.07
query4	1.63	0.10	0.10
query5	0.52	0.53	0.54
query6	1.13	0.73	0.73
query7	0.03	0.02	0.02
query8	0.06	0.04	0.03
query9	0.56	0.51	0.50
query10	0.55	0.55	0.56
query11	0.14	0.10	0.10
query12	0.14	0.12	0.11
query13	0.59	0.60	0.60
query14	2.92	3.07	2.94
query15	0.89	0.82	0.82
query16	0.36	0.37	0.38
query17	0.99	1.05	0.98
query18	0.23	0.21	0.21
query19	1.88	1.82	1.94
query20	0.01	0.01	0.01
query21	15.37	0.60	0.55
query22	2.51	1.76	1.55
query23	17.26	0.82	0.67
query24	2.85	1.31	0.75
query25	0.22	0.16	0.16
query26	0.37	0.14	0.14
query27	0.06	0.04	0.04
query28	10.86	1.09	1.07
query29	12.55	3.30	3.27
query30	0.24	0.05	0.06
query31	2.86	0.39	0.38
query32	3.23	0.46	0.47
query33	2.95	3.07	3.03
query34	16.94	4.46	4.50
query35	4.58	4.59	4.56
query36	0.68	0.49	0.51
query37	0.09	0.06	0.07
query38	0.05	0.04	0.03
query39	0.03	0.03	0.02
query40	0.16	0.13	0.13
query41	0.08	0.02	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.02
Total cold run time: 106.98 s
Total hot run time: 32.46 s

@morrySnow morrySnow merged commit d0f76d8 into branch-3.0 Jan 8, 2025
21 of 22 checks passed
@morrySnow morrySnow deleted the auto-pick-46551-branch-3.0 branch January 8, 2025 06:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@Thearas @doris-robot @dataroaring @morrySnow @englefly