Skip to content

Conversation

@Jibing-Li
Copy link
Contributor

@Jibing-Li Jibing-Li commented Feb 8, 2025

What problem does this PR solve?

In preAggForRandomDistribution function, we need to get the MV's schema when doing direct select MV.

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Feb 8, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@Jibing-Li Jibing-Li marked this pull request as ready for review February 8, 2025 11:15
@Jibing-Li
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 31617 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 17952e73e44865ae0598395537d4a875d2b39373, data reload: false

------ Round 1 ----------------------------------
q1	17573	5226	5091	5091
q2	2045	318	171	171
q3	10530	1270	762	762
q4	10238	1019	553	553
q5	7710	2364	2330	2330
q6	185	168	137	137
q7	910	733	627	627
q8	9307	1339	1081	1081
q9	4827	4632	4824	4632
q10	6867	2334	1913	1913
q11	480	282	253	253
q12	352	358	223	223
q13	17793	3714	3153	3153
q14	221	223	213	213
q15	507	471	464	464
q16	626	614	579	579
q17	565	870	338	338
q18	6490	6184	6266	6184
q19	1647	938	531	531
q20	306	317	189	189
q21	2841	2266	1894	1894
q22	355	334	299	299
Total cold run time: 102375 ms
Total hot run time: 31617 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5165	5115	5138	5115
q2	248	327	237	237
q3	2160	2698	2289	2289
q4	1451	1922	1369	1369
q5	4236	4168	4188	4168
q6	207	167	123	123
q7	1875	1858	1760	1760
q8	2636	2656	2647	2647
q9	7171	7153	7116	7116
q10	3015	3195	2799	2799
q11	586	529	493	493
q12	683	745	628	628
q13	3487	3975	3285	3285
q14	291	298	277	277
q15	503	471	483	471
q16	650	687	650	650
q17	1170	1595	1397	1397
q18	7545	7409	7229	7229
q19	775	812	875	812
q20	1930	2029	1875	1875
q21	5474	5141	4780	4780
q22	632	560	558	558
Total cold run time: 51890 ms
Total hot run time: 50078 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 191164 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 17952e73e44865ae0598395537d4a875d2b39373, data reload: false

query1	1321	946	948	946
query2	6249	1885	1841	1841
query3	11002	4558	4577	4558
query4	55903	24691	23216	23216
query5	5072	541	483	483
query6	351	203	208	203
query7	5015	523	291	291
query8	310	236	232	232
query9	6309	2474	2490	2474
query10	414	309	244	244
query11	15155	15149	14885	14885
query12	150	106	108	106
query13	1124	540	376	376
query14	10390	6689	6610	6610
query15	188	198	184	184
query16	7012	657	515	515
query17	1124	748	576	576
query18	1498	427	314	314
query19	190	192	166	166
query20	131	128	115	115
query21	212	128	109	109
query22	4394	4559	4419	4419
query23	33764	33388	33517	33388
query24	5768	2467	2462	2462
query25	480	459	426	426
query26	678	275	166	166
query27	1691	506	341	341
query28	2719	2454	2388	2388
query29	598	561	433	433
query30	214	199	155	155
query31	881	871	769	769
query32	75	63	62	62
query33	437	383	331	331
query34	803	847	526	526
query35	811	839	754	754
query36	960	1014	917	917
query37	136	113	86	86
query38	4309	4345	4274	4274
query39	1491	1472	1487	1472
query40	199	118	105	105
query41	75	49	48	48
query42	131	111	108	108
query43	488	526	484	484
query44	1349	813	841	813
query45	181	176	173	173
query46	897	1120	701	701
query47	1811	1861	1798	1798
query48	393	419	312	312
query49	698	526	433	433
query50	714	760	439	439
query51	4264	4362	4241	4241
query52	111	104	96	96
query53	236	274	193	193
query54	514	526	447	447
query55	85	80	82	80
query56	291	281	266	266
query57	1208	1211	1118	1118
query58	259	255	257	255
query59	2834	2945	2717	2717
query60	303	310	284	284
query61	142	146	141	141
query62	753	726	706	706
query63	237	200	205	200
query64	1763	1143	667	667
query65	3268	3191	3163	3163
query66	711	388	302	302
query67	15774	15759	15431	15431
query68	5444	787	530	530
query69	515	314	261	261
query70	1200	1124	1084	1084
query71	454	296	270	270
query72	6384	3849	3808	3808
query73	1274	765	362	362
query74	9146	9218	8984	8984
query75	3192	3165	2722	2722
query76	3908	1193	760	760
query77	539	382	278	278
query78	10116	10077	9341	9341
query79	2327	807	604	604
query80	874	543	450	450
query81	544	279	242	242
query82	418	156	114	114
query83	183	175	158	158
query84	301	100	76	76
query85	745	342	307	307
query86	409	305	279	279
query87	4490	4529	4502	4502
query88	3138	2218	2198	2198
query89	394	316	277	277
query90	1839	196	195	195
query91	136	136	107	107
query92	77	57	61	57
query93	2762	995	594	594
query94	676	406	306	306
query95	352	268	265	265
query96	481	569	276	276
query97	2766	2805	2716	2716
query98	222	212	199	199
query99	1304	1383	1308	1308
Total cold run time: 295628 ms
Total hot run time: 191164 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.49 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 17952e73e44865ae0598395537d4a875d2b39373, data reload: false

query1	0.03	0.03	0.03
query2	0.08	0.03	0.04
query3	0.24	0.06	0.06
query4	1.63	0.10	0.10
query5	0.42	0.41	0.40
query6	1.16	0.64	0.66
query7	0.02	0.01	0.02
query8	0.03	0.03	0.03
query9	0.61	0.53	0.51
query10	0.58	0.58	0.58
query11	0.15	0.10	0.11
query12	0.14	0.11	0.11
query13	0.62	0.60	0.61
query14	2.82	2.80	2.82
query15	0.92	0.87	0.85
query16	0.39	0.38	0.39
query17	1.07	1.05	0.99
query18	0.21	0.20	0.20
query19	1.89	1.80	2.00
query20	0.02	0.01	0.01
query21	15.39	0.93	0.57
query22	0.74	1.18	0.78
query23	14.79	1.38	0.60
query24	7.52	1.52	0.55
query25	0.53	0.23	0.20
query26	0.62	0.16	0.13
query27	0.05	0.05	0.05
query28	9.09	0.89	0.43
query29	12.67	3.98	3.27
query30	0.24	0.08	0.06
query31	2.83	0.59	0.38
query32	3.24	0.54	0.47
query33	2.99	2.97	3.07
query34	15.86	5.10	4.49
query35	4.52	4.56	4.52
query36	0.66	0.49	0.50
query37	0.09	0.06	0.05
query38	0.05	0.04	0.04
query39	0.02	0.02	0.02
query40	0.17	0.12	0.14
query41	0.08	0.02	0.02
query42	0.03	0.03	0.02
query43	0.03	0.03	0.03
Total cold run time: 105.24 s
Total hot run time: 30.49 s

@Jibing-Li
Copy link
Contributor Author

run buildall

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Feb 10, 2025
@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@doris-robot
Copy link

TPC-H: Total hot run time: 32314 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 18888f55e81e6275f3fab89f00678bf38f9c7ce2, data reload: false

------ Round 1 ----------------------------------
q1	17599	5395	5257	5257
q2	2042	309	164	164
q3	10399	1381	737	737
q4	10219	1060	542	542
q5	7970	2587	2594	2587
q6	209	180	135	135
q7	978	792	603	603
q8	9328	1548	1296	1296
q9	5164	4704	4752	4704
q10	6864	2345	1874	1874
q11	507	283	262	262
q12	351	382	219	219
q13	17768	3788	3096	3096
q14	236	226	213	213
q15	531	471	463	463
q16	644	615	594	594
q17	600	915	347	347
q18	6837	6382	6106	6106
q19	1081	1092	591	591
q20	325	347	193	193
q21	3106	2362	2029	2029
q22	370	326	302	302
Total cold run time: 103128 ms
Total hot run time: 32314 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5418	5404	5448	5404
q2	246	342	241	241
q3	2232	2759	2291	2291
q4	1482	1912	1371	1371
q5	4312	4156	4233	4156
q6	243	175	125	125
q7	2037	1962	1771	1771
q8	2795	2934	2865	2865
q9	7111	7042	7052	7042
q10	3070	3291	2790	2790
q11	636	533	489	489
q12	690	783	622	622
q13	3539	3984	3249	3249
q14	295	294	280	280
q15	526	481	460	460
q16	644	706	647	647
q17	1205	1743	1432	1432
q18	7545	7329	7299	7299
q19	870	807	929	807
q20	1990	2048	1925	1925
q21	5677	5053	4817	4817
q22	637	579	549	549
Total cold run time: 53200 ms
Total hot run time: 50632 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 183409 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 18888f55e81e6275f3fab89f00678bf38f9c7ce2, data reload: false

query1	965	381	367	367
query2	6517	1940	1851	1851
query3	6809	216	213	213
query4	26942	23374	23260	23260
query5	4394	686	485	485
query6	297	184	180	180
query7	4602	493	302	302
query8	285	234	231	231
query9	8580	2467	2477	2467
query10	455	325	266	266
query11	15549	15142	14952	14952
query12	157	110	101	101
query13	1636	516	391	391
query14	9372	6155	6229	6155
query15	202	195	175	175
query16	7164	627	446	446
query17	1186	683	537	537
query18	1929	389	290	290
query19	196	223	150	150
query20	120	115	108	108
query21	211	120	103	103
query22	4121	4369	3988	3988
query23	33790	33185	33010	33010
query24	7817	2410	2384	2384
query25	543	437	381	381
query26	1239	277	152	152
query27	2443	462	334	334
query28	4270	2391	2392	2391
query29	800	563	444	444
query30	242	188	155	155
query31	964	848	762	762
query32	75	69	64	64
query33	574	367	312	312
query34	805	847	487	487
query35	807	820	739	739
query36	969	965	907	907
query37	126	102	82	82
query38	4104	4159	4143	4143
query39	1439	1582	1400	1400
query40	217	126	113	113
query41	69	59	58	58
query42	121	108	110	108
query43	516	518	484	484
query44	1322	790	778	778
query45	201	170	167	167
query46	845	1057	666	666
query47	1716	1765	1687	1687
query48	382	418	321	321
query49	813	513	441	441
query50	684	732	448	448
query51	4150	4195	4133	4133
query52	113	106	99	99
query53	229	256	225	225
query54	489	494	406	406
query55	79	79	80	79
query56	259	286	251	251
query57	1110	1111	1065	1065
query58	247	228	236	228
query59	2713	2757	2584	2584
query60	279	272	289	272
query61	120	114	112	112
query62	850	719	643	643
query63	225	198	192	192
query64	4436	981	638	638
query65	3270	3167	3156	3156
query66	1142	402	323	323
query67	15735	15743	15593	15593
query68	8018	765	492	492
query69	475	297	265	265
query70	1213	1149	1117	1117
query71	417	304	268	268
query72	5778	3582	3828	3582
query73	747	747	346	346
query74	9171	8814	8743	8743
query75	3246	3175	2702	2702
query76	3229	1167	761	761
query77	524	373	284	284
query78	9956	10075	9313	9313
query79	2469	803	596	596
query80	687	525	448	448
query81	524	271	240	240
query82	483	151	117	117
query83	170	179	164	164
query84	244	91	78	78
query85	785	333	289	289
query86	372	296	278	278
query87	4379	4521	4470	4470
query88	3889	2172	2161	2161
query89	385	320	289	289
query90	1889	192	192	192
query91	132	139	116	116
query92	77	59	59	59
query93	2193	1000	571	571
query94	676	397	302	302
query95	359	269	263	263
query96	486	554	266	266
query97	2831	2820	2724	2724
query98	252	201	194	194
query99	1324	1365	1284	1284
Total cold run time: 272408 ms
Total hot run time: 183409 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.04 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 18888f55e81e6275f3fab89f00678bf38f9c7ce2, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.03
query3	0.23	0.06	0.07
query4	1.62	0.11	0.09
query5	0.42	0.41	0.41
query6	1.16	0.66	0.65
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.60	0.50	0.51
query10	0.56	0.58	0.56
query11	0.14	0.10	0.11
query12	0.15	0.11	0.12
query13	0.62	0.60	0.60
query14	2.72	2.73	2.80
query15	0.93	0.84	0.85
query16	0.37	0.37	0.41
query17	1.01	1.00	1.00
query18	0.21	0.19	0.19
query19	1.87	1.80	2.05
query20	0.01	0.01	0.01
query21	15.36	0.91	0.56
query22	0.73	1.31	0.65
query23	14.90	1.33	0.58
query24	6.60	1.49	1.39
query25	0.50	0.32	0.08
query26	0.54	0.15	0.14
query27	0.05	0.06	0.05
query28	10.28	0.84	0.43
query29	12.54	3.95	3.30
query30	0.25	0.08	0.06
query31	2.82	0.58	0.39
query32	3.22	0.54	0.46
query33	2.97	2.99	3.09
query34	15.86	5.05	4.52
query35	4.53	4.52	4.56
query36	0.68	0.49	0.48
query37	0.09	0.07	0.06
query38	0.05	0.03	0.04
query39	0.03	0.03	0.02
query40	0.16	0.14	0.13
query41	0.07	0.02	0.02
query42	0.03	0.02	0.03
query43	0.04	0.03	0.03
Total cold run time: 105.08 s
Total hot run time: 31.04 s

@Jibing-Li
Copy link
Contributor Author

run p0

@Jibing-Li Jibing-Li merged commit ac264d7 into apache:master Feb 10, 2025
29 of 32 checks passed
@Jibing-Li Jibing-Li deleted the directmv branch February 10, 2025 09:33
github-actions bot pushed a commit that referenced this pull request Feb 10, 2025
### What problem does this PR solve?

In preAggForRandomDistribution function, we need to get the MV's schema
when doing direct select MV.
github-actions bot pushed a commit that referenced this pull request Feb 10, 2025
### What problem does this PR solve?

In preAggForRandomDistribution function, we need to get the MV's schema
when doing direct select MV.
Jibing-Li added a commit that referenced this pull request Feb 10, 2025
Cherry-picked from #47658

Co-authored-by: James <lijibing@selectdb.com>
Jibing-Li added a commit that referenced this pull request Feb 10, 2025
Cherry-picked from #47658

Co-authored-by: James <lijibing@selectdb.com>
lzyy2024 pushed a commit to lzyy2024/doris that referenced this pull request Feb 21, 2025
### What problem does this PR solve?

In preAggForRandomDistribution function, we need to get the MV's schema
when doing direct select MV.
@yiguolei yiguolei mentioned this pull request Mar 25, 2025
koarz pushed a commit to koarz/doris that referenced this pull request Jun 4, 2025
### What problem does this PR solve?

In preAggForRandomDistribution function, we need to get the MV's schema
when doing direct select MV.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.9-merged dev/3.0.4-merged reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants