Skip to content

[opt](create table) Fixed table creation becomes slower as the number of tablets increases #52688

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
gavinchou merged 1 commit into from
Jul 8, 2025
Merged

[opt](create table) Fixed table creation becomes slower as the number of tablets increases #52688

gavinchou merged 1 commit into from
Jul 8, 2025

Conversation

@deardeng
Copy link
Contributor

@deardeng deardeng commented Jul 2, 2025 •
edited
Loading

…of tablets increases

What problem does this PR solve?

image

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

hello-stephen commented Jul 2, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@deardeng
Copy link
Contributor Author

deardeng commented Jul 2, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Jul 2, 2025

TPC-H: Total hot run time: 34161 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 16a0ef0852f04930abb08f3d3cd69403f6b55216, data reload: false

------ Round 1 ----------------------------------
q1	17612	5537	5058	5058
q2	1945	269	177	177
q3	10515	1303	734	734
q4	10315	1040	518	518
q5	8706	2410	2354	2354
q6	178	164	130	130
q7	902	730	599	599
q8	9313	1289	1138	1138
q9	7067	5228	5181	5181
q10	6944	2378	1964	1964
q11	470	293	268	268
q12	356	348	214	214
q13	17763	3645	3109	3109
q14	238	229	209	209
q15	563	476	487	476
q16	427	417	369	369
q17	587	855	358	358
q18	7319	7259	7109	7109
q19	1126	946	571	571
q20	338	339	212	212
q21	3699	2570	2431	2431
q22	1094	1032	982	982
Total cold run time: 107477 ms
Total hot run time: 34161 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5104	5073	5098	5073
q2	242	329	217	217
q3	2178	2670	2279	2279
q4	1362	1810	1352	1352
q5	4261	4583	4579	4579
q6	204	163	126	126
q7	2035	1966	1802	1802
q8	2680	2712	2577	2577
q9	7215	7315	7331	7315
q10	3192	3306	2883	2883
q11	578	528	499	499
q12	672	795	651	651
q13	3603	3957	3353	3353
q14	299	305	294	294
q15	529	494	477	477
q16	448	486	457	457
q17	1155	1554	1434	1434
q18	7927	7573	7726	7573
q19	826	816	866	816
q20	1994	2060	1973	1973
q21	4962	4537	4523	4523
q22	1123	1134	1111	1111
Total cold run time: 52589 ms
Total hot run time: 51364 ms

@doris-robot
Copy link

doris-robot commented Jul 2, 2025

TPC-DS: Total hot run time: 185660 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 16a0ef0852f04930abb08f3d3cd69403f6b55216, data reload: false

query1	996	389	379	379
query2	6557	1704	1691	1691
query3	6735	214	208	208
query4	27171	23197	23439	23197
query5	4352	579	443	443
query6	303	224	207	207
query7	4634	499	276	276
query8	264	218	206	206
query9	8613	2639	2626	2626
query10	471	322	271	271
query11	15626	14962	14886	14886
query12	155	122	103	103
query13	1637	518	391	391
query14	8478	5616	5534	5534
query15	221	182	166	166
query16	7108	662	492	492
query17	936	708	565	565
query18	1963	388	302	302
query19	186	187	156	156
query20	114	121	125	121
query21	243	120	106	106
query22	4389	4901	4853	4853
query23	34940	33772	33199	33199
query24	8485	2335	2393	2335
query25	518	455	370	370
query26	1232	257	139	139
query27	2787	507	339	339
query28	4304	2139	2112	2112
query29	745	555	435	435
query30	278	214	190	190
query31	907	811	785	785
query32	70	65	64	64
query33	571	367	308	308
query34	810	890	530	530
query35	767	822	737	737
query36	950	957	876	876
query37	110	101	81	81
query38	4175	4120	4061	4061
query39	1439	1497	1383	1383
query40	209	118	115	115
query41	61	57	55	55
query42	126	106	113	106
query43	481	514	473	473
query44	1307	849	815	815
query45	178	173	168	168
query46	857	1004	631	631
query47	1771	1789	1746	1746
query48	398	448	323	323
query49	750	514	381	381
query50	620	674	396	396
query51	4132	4198	4064	4064
query52	112	108	98	98
query53	218	243	184	184
query54	576	559	499	499
query55	77	76	83	76
query56	296	293	291	291
query57	1191	1183	1135	1135
query58	263	258	256	256
query59	2554	2687	2613	2613
query60	316	319	301	301
query61	125	122	124	122
query62	835	722	646	646
query63	219	184	178	178
query64	4272	977	649	649
query65	4294	4185	4161	4161
query66	1123	403	297	297
query67	15992	15624	15509	15509
query68	7709	880	521	521
query69	499	351	275	275
query70	1177	1112	1085	1085
query71	454	317	284	284
query72	5821	4864	4854	4854
query73	693	659	349	349
query74	8859	9148	8641	8641
query75	3900	3191	2678	2678
query76	3649	1145	703	703
query77	776	383	307	307
query78	10058	10214	9378	9378
query79	2446	826	585	585
query80	655	507	493	493
query81	493	260	219	219
query82	446	123	96	96
query83	259	251	238	238
query84	289	105	95	95
query85	792	354	314	314
query86	385	300	268	268
query87	4436	4492	4337	4337
query88	3662	2276	2255	2255
query89	387	306	287	287
query90	1834	201	205	201
query91	135	139	110	110
query92	76	59	62	59
query93	1996	923	583	583
query94	672	424	344	344
query95	366	282	280	280
query96	493	567	276	276
query97	2747	2767	2665	2665
query98	237	213	202	202
query99	1335	1412	1267	1267
Total cold run time: 275329 ms
Total hot run time: 185660 ms

@doris-robot
Copy link

doris-robot commented Jul 2, 2025

ClickBench: Total hot run time: 29.48 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 16a0ef0852f04930abb08f3d3cd69403f6b55216, data reload: false

query1	0.05	0.04	0.03
query2	0.07	0.04	0.04
query3	0.24	0.08	0.07
query4	1.62	0.11	0.11
query5	0.44	0.43	0.43
query6	1.15	0.65	0.65
query7	0.03	0.01	0.01
query8	0.04	0.03	0.04
query9	0.60	0.52	0.53
query10	0.56	0.57	0.57
query11	0.16	0.11	0.11
query12	0.15	0.12	0.12
query13	0.64	0.61	0.61
query14	0.78	0.83	0.81
query15	0.92	0.87	0.87
query16	0.38	0.39	0.39
query17	1.03	1.07	1.06
query18	0.22	0.20	0.20
query19	1.96	1.84	1.96
query20	0.01	0.01	0.02
query21	15.36	0.91	0.54
query22	0.74	1.15	0.98
query23	14.74	1.37	0.60
query24	7.64	0.68	0.51
query25	0.48	0.26	0.08
query26	0.62	0.17	0.14
query27	0.07	0.06	0.05
query28	8.81	0.92	0.44
query29	12.58	3.97	3.29
query30	0.25	0.09	0.07
query31	2.86	0.60	0.38
query32	3.25	0.55	0.47
query33	2.98	3.10	3.05
query34	16.04	5.40	4.79
query35	4.85	4.79	4.92
query36	0.69	0.51	0.48
query37	0.09	0.07	0.06
query38	0.05	0.04	0.04
query39	0.03	0.03	0.02
query40	0.18	0.14	0.14
query41	0.08	0.03	0.02
query42	0.04	0.02	0.02
query43	0.03	0.04	0.04
Total cold run time: 103.51 s
Total hot run time: 29.48 s

@hello-stephen
Copy link
Contributor

hello-stephen commented Jul 2, 2025

FE UT Coverage Report

Increment line coverage 87.50% (7/8) 🎉
Increment coverage report
Complete coverage report

dataroaring
dataroaring approved these changes Jul 8, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 8, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Jul 8, 2025

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Jul 8, 2025

PR approved by anyone and no changes requested.

@gavinchou gavinchou changed the title [fix](create table)Fixed table creation becomes slower as the number … [fix](create table)Fixed table creation becomes slower as the number of tablets increases Jul 8, 2025
@gavinchou gavinchou changed the title [fix](create table)Fixed table creation becomes slower as the number of tablets increases [opt](create table) Fixed table creation becomes slower as the number of tablets increases Jul 8, 2025
@gavinchou gavinchou merged commit bcdc287 into apache:master Jul 8, 2025
29 of 31 checks passed
github-actions bot pushed a commit that referenced this pull request Jul 8, 2025
@deardeng
[opt](create table) Fixed table creation becomes slower as the number…
44dbb25 
… of tablets increases (#52688)

<img width="508" alt="image"
src="https://github.com/user-attachments/assets/3885dbc1-6cdb-46da-85e6-2e48a1c59888"
/>
@github-actions github-actions bot mentioned this pull request Jul 8, 2025
Merged
github-actions bot pushed a commit that referenced this pull request Jul 8, 2025
@deardeng
[opt](create table) Fixed table creation becomes slower as the number…
aed6bd5 
… of tablets increases (#52688)

<img width="508" alt="image"
src="https://github.com/user-attachments/assets/3885dbc1-6cdb-46da-85e6-2e48a1c59888"
/>
@github-actions github-actions bot mentioned this pull request Jul 8, 2025
Merged
morrySnow pushed a commit that referenced this pull request Jul 8, 2025
@github-actions @deardeng
branch-3.1: [opt](create table) Fixed table creation becomes slower a…
f8e7ef2 
…s the number of tablets increases #52688 (#52919)

Cherry-picked from #52688

Co-authored-by: deardeng <dengxin@selectdb.com>
dataroaring pushed a commit that referenced this pull request Jul 9, 2025
@github-actions @deardeng
branch-3.0: [opt](create table) Fixed table creation becomes slower a…
177e4dc 
…s the number of tablets increases #52688 (#52918)

Cherry-picked from #52688

Co-authored-by: deardeng <dengxin@selectdb.com>
github-actions bot pushed a commit that referenced this pull request Jul 9, 2025
@deardeng
[opt](create table) Fixed table creation becomes slower as the number…
03dccda 
… of tablets increases (#52688)

<img width="508" alt="image"
src="https://github.com/user-attachments/assets/3885dbc1-6cdb-46da-85e6-2e48a1c59888"
/>
@github-actions github-actions bot mentioned this pull request Jul 9, 2025
Open
@dataroaring dataroaring added the usercase Important user case type label label Jul 23, 2025
yiguolei pushed a commit that referenced this pull request Jul 26, 2025
@deardeng @yiguolei
[opt](create table) Fixed table creation becomes slower as the number…
9285f3a 
… of tablets increases (#52688)

<img width="508" alt="image"
src="https://github.com/user-attachments/assets/3885dbc1-6cdb-46da-85e6-2e48a1c59888"
/>
@gavinchou gavinchou mentioned this pull request Aug 15, 2025
Open
deardeng added a commit to deardeng/incubator-doris that referenced this pull request Dec 19, 2025
@deardeng
[opt](create table) Fixed table creation becomes slower as the number…
74f23d7 
… of tablets increases (apache#52688)

<img width="508" alt="image"
src="https://github.com/user-attachments/assets/3885dbc1-6cdb-46da-85e6-2e48a1c59888"
/>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gavinchou gavinchou gavinchou approved these changes

@dataroaring dataroaring dataroaring approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.x dev/3.0.7-merged dev/3.1.0-merged reviewed usercase Important user case type label

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@deardeng @hello-stephen @doris-robot @gavinchou @dataroaring @morrySnow