Skip to content

[fix](dynamic partition) fix create too many dynamic partitions #30994

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dataroaring merged 2 commits into from
Feb 18, 2024
Merged

[fix](dynamic partition) fix create too many dynamic partitions #30994

dataroaring merged 2 commits into from
Feb 18, 2024

Conversation

@yujun777
Copy link
Contributor

@yujun777 yujun777 commented Feb 8, 2024 •
edited
Loading

For dynamic partitions, the expect created partition num = dynamic_partition.end - dynamic_partition.start.
By default, fe limit 500(Config.max_dynamic_partition_num) dynamic partitions.

But if dynamic_partition.start is near int.min_value， the expect create partition num will be overflow and become a negative number. Then check created partition num < Config.max_dynamic_partition_num is ok.

For example: if start = -2147483647 (int.min_value + 1)， end = 1, then end - start = -2147483648 < 0

So change the expect create partition num to long to avoid overflow.

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

doris-robot commented Feb 8, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

@yujun777
Copy link
Contributor Author

yujun777 commented Feb 8, 2024

run buildall

@yujun777
Copy link
Contributor Author

yujun777 commented Feb 8, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Feb 8, 2024

TPC-H: Total hot run time: 41152 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c930391a084104cc289f820915012586e7bcb451, data reload: false

------ Round 1 ----------------------------------
q1	17618	4927	4838	4838
q2	2046	149	137	137
q3	10614	997	999	997
q4	4776	947	964	947
q5	7698	3184	3216	3184
q6	202	136	130	130
q7	1289	788	751	751
q8	9339	2061	2065	2061
q9	7608	6639	6681	6639
q10	8308	2657	2643	2643
q11	409	210	203	203
q12	749	325	324	324
q13	17994	3686	3676	3676
q14	288	258	253	253
q15	591	511	503	503
q16	501	402	409	402
q17	914	862	770	770
q18	7490	6592	6609	6592
q19	1560	1484	1487	1484
q20	595	357	340	340
q21	7007	3962	3933	3933
q22	865	349	345	345
Total cold run time: 108461 ms
Total hot run time: 41152 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4844	4820	4832	4820
q2	297	179	181	179
q3	3582	3555	3570	3555
q4	2504	2514	2505	2505
q5	5776	5752	5740	5740
q6	209	128	128	128
q7	2231	1655	1636	1636
q8	3003	3064	3091	3064
q9	8687	8677	8722	8677
q10	6756	4211	4201	4201
q11	506	377	359	359
q12	760	527	539	527
q13	4990	3374	3377	3374
q14	261	248	245	245
q15	593	493	488	488
q16	488	437	433	433
q17	1656	1593	1598	1593
q18	8344	7580	7681	7580
q19	1635	1640	1635	1635
q20	2111	1826	1825	1825
q21	6518	6093	6087	6087
q22	582	524	504	504
Total cold run time: 66333 ms
Total hot run time: 59155 ms

@doris-robot
Copy link

doris-robot commented Feb 8, 2024

TPC-DS: Total hot run time: 188659 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c930391a084104cc289f820915012586e7bcb451, data reload: false

query1	931	345	341	341
query2	6538	1818	1797	1797
query3	6698	206	198	198
query4	24640	22530	22402	22402
query5	4256	376	382	376
query6	270	163	164	163
query7	4612	289	291	289
query8	252	196	196	196
query9	8438	2809	2803	2803
query10	416	237	230	230
query11	16254	15629	15450	15450
query12	133	80	86	80
query13	1665	409	401	401
query14	9059	7631	7638	7631
query15	212	180	195	180
query16	7436	254	244	244
query17	1429	561	532	532
query18	1947	264	257	257
query19	191	144	144	144
query20	86	80	81	80
query21	190	121	114	114
query22	5075	4932	4828	4828
query23	32611	31551	31684	31551
query24	12788	3384	3419	3384
query25	647	375	355	355
query26	1824	150	155	150
query27	3060	322	306	306
query28	6596	1834	1823	1823
query29	1132	609	618	609
query30	275	134	154	134
query31	942	761	773	761
query32	95	57	54	54
query33	724	226	235	226
query34	1074	486	492	486
query35	936	844	824	824
query36	996	931	916	916
query37	131	61	61	61
query38	3477	3337	3312	3312
query39	1371	1319	1325	1319
query40	284	105	101	101
query41	36	34	35	34
query42	104	102	107	102
query43	485	490	454	454
query44	1055	684	698	684
query45	197	183	182	182
query46	1037	762	782	762
query47	1735	1614	1634	1614
query48	414	355	345	345
query49	1165	297	296	296
query50	794	371	385	371
query51	5329	5158	5151	5151
query52	106	88	88	88
query53	393	286	296	286
query54	299	221	228	221
query55	84	82	84	82
query56	210	200	201	200
query57	1089	972	993	972
query58	217	194	195	194
query59	2467	2281	2279	2279
query60	233	214	217	214
query61	82	82	88	82
query62	610	382	345	345
query63	323	282	281	281
query64	6383	3124	3075	3075
query65	3314	3267	3265	3265
query66	1348	322	325	322
query67	14570	14210	14165	14165
query68	5151	546	540	540
query69	518	356	351	351
query70	1286	1236	1208	1208
query71	442	256	255	255
query72	6281	2786	2601	2601
query73	713	311	308	308
query74	7139	6700	6692	6692
query75	3216	2580	2549	2549
query76	3257	1128	1192	1128
query77	362	234	231	231
query78	9403	8832	8830	8830
query79	994	518	510	510
query80	657	352	335	335
query81	440	208	207	207
query82	1301	84	85	84
query83	239	125	119	119
query84	227	77	78	77
query85	1158	339	324	324
query86	311	307	304	304
query87	3645	3403	3390	3390
query88	2721	2290	2285	2285
query89	435	355	352	352
query90	2094	165	168	165
query91	155	124	128	124
query92	52	49	48	48
query93	1021	487	491	487
query94	1117	187	183	183
query95	8459	8331	8494	8331
query96	574	270	267	267
query97	4467	4277	4295	4277
query98	220	205	192	192
query99	1101	738	731	731
Total cold run time: 282926 ms
Total hot run time: 188659 ms

@doris-robot
Copy link

doris-robot commented Feb 8, 2024

ClickBench: Total hot run time: 31 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c930391a084104cc289f820915012586e7bcb451, data reload: false

query1	0.02	0.02	0.02
query2	0.06	0.02	0.02
query3	0.23	0.07	0.08
query4	1.64	0.08	0.08
query5	0.49	0.49	0.48
query6	1.36	0.60	0.60
query7	0.01	0.01	0.02
query8	0.03	0.02	0.02
query9	0.53	0.45	0.46
query10	0.49	0.48	0.49
query11	0.12	0.10	0.10
query12	0.11	0.10	0.10
query13	0.58	0.59	0.59
query14	0.76	0.77	0.80
query15	0.82	0.81	0.79
query16	0.34	0.33	0.34
query17	0.88	0.90	0.93
query18	0.17	0.17	0.15
query19	1.79	1.70	1.68
query20	0.02	0.01	0.01
query21	15.41	0.58	0.59
query22	2.90	3.35	2.34
query23	17.31	1.09	1.12
query24	1.98	0.59	0.37
query25	0.65	0.06	0.05
query26	0.19	0.13	0.15
query27	0.06	0.06	0.05
query28	11.99	0.83	0.82
query29	12.60	3.42	3.35
query30	0.53	0.51	0.44
query31	2.77	0.36	0.36
query32	3.34	0.47	0.47
query33	3.13	3.13	3.11
query34	15.35	4.48	4.49
query35	4.51	4.49	4.49
query36	1.07	0.93	0.94
query37	0.07	0.05	0.06
query38	0.03	0.03	0.03
query39	0.02	0.02	0.02
query40	0.18	0.14	0.14
query41	0.07	0.01	0.02
query42	0.02	0.02	0.02
query43	0.02	0.02	0.02
Total cold run time: 104.65 s
Total hot run time: 31 s

@doris-robot
Copy link

doris-robot commented Feb 8, 2024

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit c930391a084104cc289f820915012586e7bcb451 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       14.4 seconds inserted 10000000 Rows, about 694K ops/s

dataroaring
dataroaring approved these changes Feb 18, 2024
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Feb 18, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Feb 18, 2024

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Feb 18, 2024

PR approved by anyone and no changes requested.

wm1581066
wm1581066 approved these changes Feb 18, 2024
View reviewed changes
Copy link
Collaborator

@wm1581066 wm1581066 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit 6478a6c into apache:master Feb 18, 2024
yujun777 added a commit to yujun777/doris that referenced this pull request Feb 18, 2024
@yujun777
[fix](dynamic partition) fix create too many dynamic partitions (apac…
f6eb57e 
…he#30994)
yujun777 added a commit to yujun777/doris that referenced this pull request Feb 18, 2024
@yujun777
[fix](dynamic partition) fix create too many dynamic partitions (apac…
af9710e 
…he#30994)
@yujun777 yujun777 mentioned this pull request Feb 18, 2024
Merged
yujun777 added a commit to yujun777/doris that referenced this pull request Feb 18, 2024
@yujun777
[fix](dynamic partition) fix create too many dynamic partitions (apac…
83eb79d 
…he#30994)
yiguolei pushed a commit that referenced this pull request Feb 18, 2024
@yujun777 @Doris-Extras
[fix](dynamic partition) fix create too many dynamic partitions (#30994)
ea4ad11
xiaokang pushed a commit that referenced this pull request Feb 18, 2024
@yujun777
[fix](dynamic partition) fix create too many dynamic partitions (#30994…
a376f0c 
…) (#31039)
@xiaokang xiaokang added usercase Important user case type label dev/2.0.5-merged and removed dev/2.0.5 labels Feb 18, 2024
mymeiyi pushed a commit to mymeiyi/doris that referenced this pull request Feb 19, 2024
@yujun777 @mymeiyi
[fix](dynamic partition) fix create too many dynamic partitions (apac…
50695cb 
…he#30994)
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
@yujun777
[fix](dynamic partition) fix create too many dynamic partitions (apac…
a97927e 
…he#30994) (apache#31039)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dataroaring dataroaring dataroaring approved these changes

+1 more reviewer

@wm1581066 wm1581066 wm1581066 approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.0.5-merged dev/2.1.0 reviewed usercase Important user case type label

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@yujun777 @doris-robot @wm1581066 @dataroaring @xiaokang