Skip to content

[Cloud](Variant) support DESC merge variants's schema for cloud mode #37955

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
eldenmoon merged 2 commits into from
Jul 19, 2024
Merged

[Cloud](Variant) support DESC merge variants's schema for cloud mode #37955

eldenmoon merged 2 commits into from
Jul 19, 2024

Conversation

@eldenmoon
Copy link
Member

@eldenmoon eldenmoon commented Jul 16, 2024

Proposed changes

Issue Number: close #xxx

@doris-robot
Copy link

doris-robot commented Jul 16, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@eldenmoon
Copy link
Member Author

eldenmoon commented Jul 16, 2024

run buildall

github-actions[bot]
github-actions bot reviewed Jul 16, 2024
View reviewed changes
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

@doris-robot
Copy link

doris-robot commented Jul 16, 2024

TPC-H: Total hot run time: 39908 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ec6cbe21652bdb8e4d234d260d939fe326a55b53, data reload: false

------ Round 1 ----------------------------------
q1	17606	4384	4272	4272
q2	2012	197	187	187
q3	10443	1253	1141	1141
q4	10204	780	736	736
q5	7536	2705	2693	2693
q6	226	139	140	139
q7	969	614	614	614
q8	9217	2078	2093	2078
q9	8775	6542	6575	6542
q10	8867	3793	3777	3777
q11	451	249	243	243
q12	460	231	225	225
q13	18962	3029	3051	3029
q14	278	236	232	232
q15	518	487	485	485
q16	485	374	390	374
q17	992	644	712	644
q18	8162	7487	7488	7487
q19	6767	1354	1246	1246
q20	720	313	318	313
q21	4960	3165	3283	3165
q22	353	298	286	286
Total cold run time: 118963 ms
Total hot run time: 39908 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4384	4266	4245	4245
q2	360	274	262	262
q3	3034	2951	2984	2951
q4	1975	1745	1743	1743
q5	5544	5555	5452	5452
q6	223	141	135	135
q7	2218	1894	1867	1867
q8	3305	3404	3422	3404
q9	8835	8892	8787	8787
q10	4136	3773	3840	3773
q11	584	494	523	494
q12	822	641	646	641
q13	15964	3191	3247	3191
q14	325	290	303	290
q15	532	487	485	485
q16	490	449	448	448
q17	1811	1527	1502	1502
q18	8267	8050	7888	7888
q19	1719	1625	1512	1512
q20	2145	1889	1854	1854
q21	5105	4913	4717	4717
q22	582	515	519	515
Total cold run time: 72360 ms
Total hot run time: 56156 ms

@doris-robot
Copy link

doris-robot commented Jul 16, 2024

TPC-DS: Total hot run time: 173587 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ec6cbe21652bdb8e4d234d260d939fe326a55b53, data reload: false

query1	915	372	361	361
query2	6454	1911	1780	1780
query3	6634	221	217	217
query4	23491	17451	17419	17419
query5	3728	485	492	485
query6	267	175	165	165
query7	4583	295	294	294
query8	240	222	184	184
query9	8590	2506	2471	2471
query10	439	276	276	276
query11	10589	10320	9958	9958
query12	112	81	82	81
query13	1627	373	359	359
query14	10405	7782	7676	7676
query15	212	162	168	162
query16	7658	314	303	303
query17	1599	566	528	528
query18	1939	274	274	274
query19	198	147	144	144
query20	90	79	82	79
query21	206	126	126	126
query22	4484	4126	4058	4058
query23	34108	33657	33596	33596
query24	10709	2949	2941	2941
query25	609	396	380	380
query26	710	156	149	149
query27	2322	280	281	280
query28	5951	2108	2101	2101
query29	918	635	678	635
query30	259	157	162	157
query31	1029	771	787	771
query32	97	55	54	54
query33	647	291	295	291
query34	892	495	494	494
query35	692	581	575	575
query36	1155	976	992	976
query37	153	86	87	86
query38	2966	2833	2817	2817
query39	892	877	855	855
query40	207	129	127	127
query41	51	46	45	45
query42	116	109	110	109
query43	523	480	454	454
query44	1090	744	729	729
query45	195	164	165	164
query46	1093	735	737	735
query47	1902	1781	1795	1781
query48	359	299	300	299
query49	832	399	418	399
query50	780	397	406	397
query51	7025	6743	6691	6691
query52	108	88	94	88
query53	356	297	295	295
query54	873	453	454	453
query55	79	73	76	73
query56	291	265	281	265
query57	1110	1041	1044	1041
query58	252	245	245	245
query59	2686	2547	2604	2547
query60	298	270	283	270
query61	118	98	98	98
query62	789	645	644	644
query63	321	293	295	293
query64	9207	2195	1690	1690
query65	3151	3117	3084	3084
query66	747	325	331	325
query67	15472	15122	14951	14951
query68	4507	539	540	539
query69	472	331	330	330
query70	1208	1047	1168	1047
query71	365	296	289	289
query72	6934	5821	5494	5494
query73	737	338	333	333
query74	6368	5631	5661	5631
query75	3410	2657	2738	2657
query76	2189	949	862	862
query77	428	306	313	306
query78	9636	9118	8939	8939
query79	2433	529	530	529
query80	2379	498	478	478
query81	610	225	230	225
query82	964	135	135	135
query83	308	169	175	169
query84	264	88	94	88
query85	1133	309	298	298
query86	448	324	322	322
query87	3290	3099	3109	3099
query88	3842	2464	2466	2464
query89	482	377	378	377
query90	1697	192	193	192
query91	128	102	105	102
query92	58	49	55	49
query93	2400	516	507	507
query94	1104	216	220	216
query95	411	325	327	325
query96	604	273	276	273
query97	3183	3054	3022	3022
query98	227	197	192	192
query99	1597	1264	1248	1248
Total cold run time: 272416 ms
Total hot run time: 173587 ms

@doris-robot
Copy link

doris-robot commented Jul 16, 2024

ClickBench: Total hot run time: 30.09 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ec6cbe21652bdb8e4d234d260d939fe326a55b53, data reload: false

query1	0.04	0.03	0.04
query2	0.08	0.03	0.04
query3	0.22	0.05	0.05
query4	1.67	0.06	0.07
query5	0.49	0.48	0.50
query6	1.14	0.73	0.73
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.55	0.49	0.49
query10	0.55	0.56	0.55
query11	0.15	0.11	0.11
query12	0.15	0.12	0.12
query13	0.60	0.59	0.58
query14	0.76	0.78	0.78
query15	0.86	0.81	0.82
query16	0.36	0.36	0.36
query17	1.00	0.96	0.99
query18	0.23	0.22	0.22
query19	1.93	1.71	1.77
query20	0.01	0.01	0.01
query21	15.43	0.73	0.65
query22	4.44	8.18	1.61
query23	18.27	1.37	1.20
query24	2.09	0.21	0.23
query25	0.16	0.09	0.09
query26	0.29	0.21	0.20
query27	0.46	0.23	0.23
query28	13.31	1.02	0.99
query29	12.66	3.26	3.31
query30	0.26	0.06	0.05
query31	2.91	0.41	0.39
query32	3.23	0.50	0.47
query33	2.87	2.86	2.92
query34	17.03	4.29	4.38
query35	4.46	4.39	4.40
query36	0.66	0.48	0.48
query37	0.18	0.15	0.15
query38	0.15	0.14	0.14
query39	0.04	0.04	0.04
query40	0.16	0.13	0.13
query41	0.09	0.05	0.05
query42	0.05	0.05	0.04
query43	0.05	0.04	0.04
Total cold run time: 110.11 s
Total hot run time: 30.09 s

xiaokang
xiaokang previously approved these changes Jul 18, 2024
View reviewed changes
Copy link
Contributor

@xiaokang xiaokang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions
Copy link
Contributor

github-actions bot commented Jul 18, 2024

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Jul 18, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Jul 18, 2024

PR approved by anyone and no changes requested.

qidaye
qidaye previously approved these changes Jul 19, 2024
View reviewed changes
Copy link
Contributor

@qidaye qidaye left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@eldenmoon eldenmoon dismissed stale reviews from qidaye and xiaokang via 2322b80 July 19, 2024 02:59
@eldenmoon
Copy link
Member Author

eldenmoon commented Jul 19, 2024

run buildall

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Jul 19, 2024
@doris-robot
Copy link

doris-robot commented Jul 19, 2024

TPC-H: Total hot run time: 40429 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2322b805d9047b5e60646cb66ae78bb32fa1c9ea, data reload: false

------ Round 1 ----------------------------------
q1	17674	4750	4472	4472
q2	2019	202	188	188
q3	10540	1219	1120	1120
q4	10206	888	854	854
q5	7656	2748	2707	2707
q6	222	141	142	141
q7	969	604	604	604
q8	9215	2125	2112	2112
q9	8756	6629	6598	6598
q10	8833	3831	3798	3798
q11	458	242	249	242
q12	421	234	233	233
q13	18687	2963	2960	2960
q14	280	231	241	231
q15	533	488	480	480
q16	498	391	376	376
q17	993	639	753	639
q18	8225	7609	7515	7515
q19	6685	1444	1345	1345
q20	669	321	315	315
q21	4953	3217	3262	3217
q22	351	286	282	282
Total cold run time: 118843 ms
Total hot run time: 40429 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4437	4365	4260	4260
q2	378	279	283	279
q3	3094	2897	2963	2897
q4	1946	1793	1768	1768
q5	5626	5556	5594	5556
q6	237	136	147	136
q7	2294	1883	1851	1851
q8	3301	3487	3488	3487
q9	8989	9122	9013	9013
q10	4152	3803	3856	3803
q11	587	510	549	510
q12	832	645	675	645
q13	16017	3178	3184	3178
q14	322	294	290	290
q15	543	494	485	485
q16	493	428	429	428
q17	1832	1557	1520	1520
q18	8265	7993	7876	7876
q19	1774	1582	1612	1582
q20	3510	1928	1887	1887
q21	11863	4800	4840	4800
q22	606	511	498	498
Total cold run time: 81098 ms
Total hot run time: 56749 ms

@doris-robot
Copy link

doris-robot commented Jul 19, 2024

TPC-DS: Total hot run time: 174541 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2322b805d9047b5e60646cb66ae78bb32fa1c9ea, data reload: false

query1	921	373	366	366
query2	6356	1898	1873	1873
query3	6632	207	217	207
query4	27902	17615	17517	17517
query5	3647	476	467	467
query6	262	198	172	172
query7	4593	292	289	289
query8	243	198	203	198
query9	8640	2403	2382	2382
query10	446	292	280	280
query11	12310	10304	10290	10290
query12	123	85	83	83
query13	1639	380	374	374
query14	10209	7616	7353	7353
query15	215	171	175	171
query16	7291	328	330	328
query17	1333	563	540	540
query18	1890	289	281	281
query19	203	155	155	155
query20	90	81	83	81
query21	208	133	124	124
query22	4405	4103	4206	4103
query23	34255	33704	33626	33626
query24	10984	2948	2916	2916
query25	630	421	426	421
query26	706	160	161	160
query27	2263	287	278	278
query28	5749	2074	2069	2069
query29	920	667	676	667
query30	258	156	160	156
query31	990	773	769	769
query32	95	60	55	55
query33	692	332	316	316
query34	883	498	508	498
query35	694	597	593	593
query36	1141	1018	986	986
query37	150	89	86	86
query38	2985	2921	3010	2921
query39	891	831	866	831
query40	197	116	117	116
query41	45	43	45	43
query42	111	98	96	96
query43	507	478	465	465
query44	1081	728	711	711
query45	193	161	160	160
query46	1101	725	730	725
query47	1879	1767	1798	1767
query48	360	290	281	281
query49	846	412	406	406
query50	779	385	389	385
query51	6796	6810	6795	6795
query52	103	91	89	89
query53	357	285	285	285
query54	860	441	432	432
query55	72	76	71	71
query56	282	261	285	261
query57	1132	1080	1038	1038
query58	260	250	260	250
query59	2866	2775	2677	2677
query60	297	273	267	267
query61	94	90	95	90
query62	788	653	647	647
query63	314	286	284	284
query64	9160	2230	1673	1673
query65	3146	3112	3151	3112
query66	784	332	331	331
query67	15736	15042	14968	14968
query68	7056	548	574	548
query69	739	446	343	343
query70	1223	1135	1088	1088
query71	509	273	280	273
query72	9093	5809	5772	5772
query73	833	322	323	322
query74	6201	5748	5768	5748
query75	4884	2688	2693	2688
query76	4527	965	959	959
query77	784	307	304	304
query78	9931	8940	8933	8933
query79	10355	531	527	527
query80	1161	491	483	483
query81	560	225	221	221
query82	696	137	138	137
query83	308	166	171	166
query84	275	89	87	87
query85	1208	307	302	302
query86	379	300	315	300
query87	3372	3156	3115	3115
query88	4638	2345	2351	2345
query89	529	386	408	386
query90	2039	197	195	195
query91	133	100	143	100
query92	67	49	52	49
query93	7135	499	493	493
query94	1158	211	210	210
query95	404	308	316	308
query96	618	280	267	267
query97	3170	3017	3047	3017
query98	241	223	205	205
query99	1522	1243	1278	1243
Total cold run time: 299738 ms
Total hot run time: 174541 ms

@doris-robot
Copy link

doris-robot commented Jul 19, 2024

ClickBench: Total hot run time: 31.59 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2322b805d9047b5e60646cb66ae78bb32fa1c9ea, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.04	0.04
query3	0.22	0.05	0.05
query4	1.68	0.07	0.07
query5	0.51	0.49	0.48
query6	1.14	0.73	0.72
query7	0.02	0.01	0.01
query8	0.04	0.04	0.04
query9	0.55	0.50	0.50
query10	0.55	0.55	0.55
query11	0.17	0.11	0.11
query12	0.15	0.12	0.13
query13	0.59	0.60	0.58
query14	0.76	0.77	0.78
query15	0.86	0.82	0.80
query16	0.38	0.35	0.36
query17	0.94	1.02	1.04
query18	0.24	0.22	0.22
query19	1.96	1.84	1.84
query20	0.02	0.01	0.01
query21	15.39	0.74	0.66
query22	3.72	6.44	2.67
query23	18.32	1.41	1.26
query24	2.11	0.24	0.21
query25	0.15	0.08	0.09
query26	0.29	0.21	0.20
query27	0.45	0.24	0.23
query28	13.22	1.02	1.00
query29	12.60	3.30	3.27
query30	0.25	0.06	0.06
query31	2.89	0.40	0.40
query32	3.26	0.46	0.47
query33	2.90	2.90	2.91
query34	17.33	4.33	4.41
query35	4.43	4.48	4.50
query36	0.66	0.48	0.49
query37	0.19	0.16	0.15
query38	0.15	0.16	0.16
query39	0.04	0.04	0.03
query40	0.14	0.12	0.12
query41	0.10	0.04	0.05
query42	0.06	0.05	0.05
query43	0.04	0.05	0.04
Total cold run time: 109.59 s
Total hot run time: 31.59 s

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 19, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Jul 19, 2024

PR approved by at least one committer and no changes requested.

csun5285
csun5285 approved these changes Jul 19, 2024
View reviewed changes
Copy link
Contributor

@csun5285 csun5285 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@eldenmoon eldenmoon merged commit bbd1f69 into apache:master Jul 19, 2024
eldenmoon added a commit to eldenmoon/incubator-doris that referenced this pull request Jul 19, 2024
@eldenmoon
[Cloud](Variant) support DESC merge variants's schema for cloud mode (a…
efa5e6d 
…pache#37955)
@eldenmoon eldenmoon mentioned this pull request Jul 19, 2024
Merged
eldenmoon added a commit that referenced this pull request Jul 19, 2024
@eldenmoon
[Cloud](Variant) support DESC merge variants's schema for cloud mode … (
25f9bc9 
#38143)

…(#37955)

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@github-actions github-actions[bot] github-actions[bot] left review comments

@qidaye qidaye qidaye approved these changes

@xiaokang xiaokang xiaokang left review comments

+1 more reviewer

@csun5285 csun5285 csun5285 approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/3.0.0-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@eldenmoon @doris-robot @xiaokang @qidaye @csun5285