Skip to content

Conversation

@wuwenchi
Copy link
Contributor

Proposed changes

Save the table schema, reduce the number of HMS calls, and improve write performance.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@wuwenchi wuwenchi changed the title [opt](hive) save table schema in transaction [opt](hive) save hive table schema in transaction Jun 28, 2024
@wuwenchi
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39880 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 78ad69360e2907b6d81935117515b68ff5e1c83b, data reload: false

------ Round 1 ----------------------------------
q1	17612	4397	4253	4253
q2	2010	188	190	188
q3	10462	1197	1129	1129
q4	10182	814	708	708
q5	7480	2670	2613	2613
q6	220	140	141	140
q7	943	600	608	600
q8	9221	2070	2038	2038
q9	8871	6489	6515	6489
q10	8961	3667	3681	3667
q11	451	246	242	242
q12	457	241	232	232
q13	17778	3014	2998	2998
q14	265	227	212	212
q15	551	479	479	479
q16	492	378	380	378
q17	965	674	647	647
q18	7951	7465	7484	7465
q19	4442	1526	1516	1516
q20	649	330	330	330
q21	4890	3214	3213	3213
q22	383	343	345	343
Total cold run time: 115236 ms
Total hot run time: 39880 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4362	4243	4238	4238
q2	382	265	259	259
q3	2998	2744	2944	2744
q4	1968	1721	1706	1706
q5	5559	5575	5498	5498
q6	220	139	147	139
q7	2240	1839	1867	1839
q8	3281	3390	3406	3390
q9	8626	8660	8797	8660
q10	4099	3903	3763	3763
q11	579	482	486	482
q12	800	654	629	629
q13	17150	3214	3169	3169
q14	304	271	278	271
q15	525	477	495	477
q16	502	438	443	438
q17	1817	1510	1496	1496
q18	8136	7826	7887	7826
q19	1788	1742	1631	1631
q20	3079	1900	1857	1857
q21	5106	5062	4810	4810
q22	740	543	551	543
Total cold run time: 74261 ms
Total hot run time: 55865 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 173666 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 78ad69360e2907b6d81935117515b68ff5e1c83b, data reload: false

query1	926	379	382	379
query2	6467	2523	2321	2321
query3	6634	210	212	210
query4	18714	17491	17399	17399
query5	3600	465	481	465
query6	265	204	163	163
query7	4601	300	302	300
query8	299	291	292	291
query9	8521	2446	2441	2441
query10	543	283	273	273
query11	10526	10043	10152	10043
query12	111	84	89	84
query13	1636	376	366	366
query14	10159	7463	6456	6456
query15	224	186	189	186
query16	7756	262	273	262
query17	1900	548	514	514
query18	1943	273	269	269
query19	193	151	153	151
query20	91	87	80	80
query21	211	124	124	124
query22	4621	4247	4273	4247
query23	33958	33751	33476	33476
query24	11166	2907	2936	2907
query25	596	381	403	381
query26	707	171	156	156
query27	2324	337	334	334
query28	6292	2199	2179	2179
query29	912	662	676	662
query30	265	166	153	153
query31	972	765	762	762
query32	95	55	51	51
query33	766	311	291	291
query34	1012	477	489	477
query35	768	644	629	629
query36	1114	1010	972	972
query37	149	78	87	78
query38	3011	2817	2852	2817
query39	894	830	841	830
query40	206	139	129	129
query41	53	55	54	54
query42	106	101	103	101
query43	595	553	558	553
query44	1123	744	758	744
query45	193	165	164	164
query46	1091	750	724	724
query47	1900	1802	1793	1793
query48	359	301	294	294
query49	836	406	408	406
query50	775	387	402	387
query51	6826	6821	6655	6655
query52	101	100	98	98
query53	363	294	297	294
query54	858	451	440	440
query55	74	78	74	74
query56	276	262	257	257
query57	1124	1066	1063	1063
query58	235	244	243	243
query59	3365	3051	3108	3051
query60	293	278	284	278
query61	95	95	89	89
query62	596	442	437	437
query63	322	293	297	293
query64	8571	2216	1731	1731
query65	3231	3137	3105	3105
query66	755	323	326	323
query67	15612	14951	14861	14861
query68	4536	551	557	551
query69	626	443	383	383
query70	1194	1075	1129	1075
query71	418	278	274	274
query72	8121	5558	5484	5484
query73	758	326	325	325
query74	5841	5504	5487	5487
query75	3520	2701	2686	2686
query76	2779	926	920	920
query77	628	303	296	296
query78	10574	9980	9643	9643
query79	2714	526	534	526
query80	2044	538	490	490
query81	565	224	220	220
query82	797	106	104	104
query83	333	176	172	172
query84	270	91	85	85
query85	1511	279	268	268
query86	455	335	319	319
query87	3353	3100	3117	3100
query88	4140	2371	2371	2371
query89	487	380	393	380
query90	1679	189	189	189
query91	130	104	101	101
query92	58	49	53	49
query93	3820	530	519	519
query94	1018	191	190	190
query95	407	317	306	306
query96	594	270	276	270
query97	3241	3055	3058	3055
query98	213	197	200	197
query99	1245	865	811	811
Total cold run time: 272945 ms
Total hot run time: 173666 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.13 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 78ad69360e2907b6d81935117515b68ff5e1c83b, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.23	0.06	0.05
query4	1.68	0.08	0.09
query5	0.50	0.49	0.50
query6	1.13	0.72	0.73
query7	0.02	0.02	0.01
query8	0.04	0.04	0.04
query9	0.54	0.49	0.49
query10	0.54	0.53	0.54
query11	0.15	0.11	0.12
query12	0.15	0.12	0.12
query13	0.60	0.59	0.60
query14	0.77	0.77	0.79
query15	0.84	0.82	0.82
query16	0.37	0.37	0.37
query17	1.02	1.07	1.03
query18	0.21	0.23	0.26
query19	1.78	1.79	1.81
query20	0.01	0.01	0.01
query21	15.44	0.73	0.65
query22	4.70	5.80	2.42
query23	18.28	1.40	1.19
query24	2.11	0.23	0.23
query25	0.16	0.08	0.08
query26	0.27	0.17	0.18
query27	0.08	0.08	0.08
query28	13.16	1.01	1.00
query29	12.63	3.27	3.23
query30	0.25	0.06	0.05
query31	2.87	0.39	0.39
query32	3.28	0.48	0.47
query33	2.91	2.90	2.90
query34	17.24	4.49	4.39
query35	4.54	4.49	4.53
query36	0.65	0.46	0.46
query37	0.17	0.15	0.16
query38	0.15	0.15	0.14
query39	0.05	0.03	0.03
query40	0.17	0.15	0.15
query41	0.10	0.06	0.05
query42	0.06	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 110.01 s
Total hot run time: 31.13 s

@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Jun 30, 2024
@github-actions
Copy link
Contributor

PR approved by anyone and no changes requested.

Copy link
Contributor

@kaka11chen kaka11chen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@morningman morningman merged commit c8f1b9f into apache:master Jul 1, 2024
dataroaring pushed a commit that referenced this pull request Jul 2, 2024
Save the table schema, reduce the number of HMS calls, and improve write
performance.
morningman pushed a commit that referenced this pull request Jul 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.5-merged dev/3.0.0-merged reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants