Skip to content

Conversation

@htyoung
Copy link
Contributor

@htyoung htyoung commented Aug 21, 2024

Proposed changes

Issue Number: close #39684

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@htyoung htyoung force-pushed the change-list-partition-buckets branch from af5d6b7 to 116f470 Compare August 21, 2024 10:31
@htyoung htyoung changed the title [Fix](fe) LIST partition table support modify default bucket num [fix](fe) LIST partition table support modify default bucket num Aug 21, 2024
@htyoung
Copy link
Contributor Author

htyoung commented Aug 21, 2024

run buildall

morningman
morningman previously approved these changes Aug 21, 2024
Copy link
Contributor

@morningman morningman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Aug 21, 2024
@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@doris-robot
Copy link

TPC-H: Total hot run time: 38634 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 116f4703def4e9c14a13e1b9467e868f30ae5a2f, data reload: false

------ Round 1 ----------------------------------
q1	17866	4434	4294	4294
q2	2032	221	222	221
q3	11976	982	1075	982
q4	10549	667	806	667
q5	7800	2825	2841	2825
q6	266	157	157	157
q7	1012	657	652	652
q8	9592	2107	2144	2107
q9	7848	6554	6589	6554
q10	7080	2292	2246	2246
q11	544	279	274	274
q12	432	260	264	260
q13	18430	3043	3040	3040
q14	301	271	269	269
q15	569	535	540	535
q16	534	433	421	421
q17	990	778	725	725
q18	7348	6913	6882	6882
q19	1469	1128	1066	1066
q20	708	369	361	361
q21	4160	3034	3121	3034
q22	1162	1062	1069	1062
Total cold run time: 112668 ms
Total hot run time: 38634 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4383	4341	4300	4300
q2	413	318	310	310
q3	2908	2687	2694	2687
q4	1966	1807	1803	1803
q5	5624	5636	5736	5636
q6	245	154	147	147
q7	2275	1834	1854	1834
q8	3395	3447	3439	3439
q9	8833	8796	8793	8793
q10	3565	3289	3368	3289
q11	640	582	556	556
q12	878	682	677	677
q13	15979	3228	3171	3171
q14	330	304	299	299
q15	563	542	545	542
q16	512	478	450	450
q17	1836	1495	1499	1495
q18	8294	7884	7880	7880
q19	2043	1608	1632	1608
q20	2173	1892	1975	1892
q21	5717	5656	5573	5573
q22	1242	1114	1134	1114
Total cold run time: 73814 ms
Total hot run time: 57495 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 198569 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 116f4703def4e9c14a13e1b9467e868f30ae5a2f, data reload: false

query1	1306	919	903	903
query2	6425	2094	1988	1988
query3	10776	3981	4215	3981
query4	59847	26852	23189	23189
query5	5518	736	739	736
query6	466	219	214	214
query7	5817	325	330	325
query8	521	441	434	434
query9	8808	2548	2552	2548
query10	555	354	354	354
query11	16091	15146	15707	15146
query12	190	131	140	131
query13	1592	439	440	439
query14	11360	7577	7638	7577
query15	272	229	191	191
query16	7546	528	541	528
query17	1164	651	620	620
query18	2077	355	353	353
query19	318	173	184	173
query20	157	140	141	140
query21	249	146	155	146
query22	4685	4581	4376	4376
query23	34449	33961	33903	33903
query24	6199	3049	2967	2967
query25	573	426	433	426
query26	703	181	185	181
query27	1768	317	310	310
query28	3837	2168	2159	2159
query29	742	466	454	454
query30	234	203	199	199
query31	1078	862	869	862
query32	109	77	81	77
query33	493	358	362	358
query34	955	541	524	524
query35	890	779	768	768
query36	1077	987	978	978
query37	159	110	107	107
query38	3975	3869	3878	3869
query39	1514	1455	1475	1455
query40	236	158	157	157
query41	141	138	142	138
query42	132	116	118	116
query43	559	517	520	517
query44	1140	785	805	785
query45	228	202	195	195
query46	1141	762	760	760
query47	1976	1842	1847	1842
query48	414	349	345	345
query49	940	580	580	580
query50	894	470	469	469
query51	7141	7148	7097	7097
query52	118	107	107	107
query53	301	224	224	224
query54	617	512	514	512
query55	92	93	92	92
query56	348	320	324	320
query57	1282	1141	1141	1141
query58	306	324	295	295
query59	3201	2964	3134	2964
query60	367	325	350	325
query61	151	150	173	150
query62	810	709	723	709
query63	260	226	230	226
query64	3230	1843	1821	1821
query65	3292	3235	3198	3198
query66	1011	668	676	668
query67	15779	15381	15335	15335
query68	4416	615	589	589
query69	473	329	336	329
query70	1198	1150	1170	1150
query71	411	319	316	316
query72	6545	2392	2099	2099
query73	805	367	365	365
query74	9235	8952	8961	8952
query75	3427	2812	2808	2808
query76	1663	1053	1034	1034
query77	661	452	453	452
query78	9736	9137	9057	9057
query79	1076	569	561	561
query80	938	615	622	615
query81	579	266	264	264
query82	311	160	162	160
query83	290	227	212	212
query84	285	97	99	97
query85	866	354	354	354
query86	348	332	329	329
query87	4480	4381	4282	4282
query88	3461	2528	2551	2528
query89	442	330	349	330
query90	1897	226	239	226
query91	150	128	129	128
query92	88	78	76	76
query93	1090	546	560	546
query94	783	325	323	323
query95	389	300	305	300
query96	607	289	293	289
query97	3232	3093	3081	3081
query98	246	226	226	226
query99	1757	1314	1327	1314
Total cold run time: 312487 ms
Total hot run time: 198569 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.81 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 116f4703def4e9c14a13e1b9467e868f30ae5a2f, data reload: false

query1	0.05	0.05	0.04
query2	0.07	0.04	0.04
query3	0.22	0.06	0.05
query4	1.67	0.08	0.10
query5	0.52	0.48	0.50
query6	1.13	0.73	0.73
query7	0.02	0.02	0.02
query8	0.06	0.05	0.04
query9	0.56	0.50	0.48
query10	0.54	0.56	0.55
query11	0.16	0.12	0.12
query12	0.16	0.13	0.13
query13	0.62	0.60	0.60
query14	0.78	0.80	0.80
query15	0.84	0.82	0.83
query16	0.37	0.37	0.39
query17	1.00	1.03	1.06
query18	0.22	0.21	0.21
query19	1.90	1.72	1.73
query20	0.02	0.01	0.01
query21	15.39	0.67	0.66
query22	4.28	6.18	2.43
query23	18.30	1.35	1.21
query24	2.08	0.22	0.24
query25	0.16	0.10	0.09
query26	0.26	0.18	0.18
query27	0.08	0.08	0.08
query28	13.28	1.02	1.01
query29	12.66	3.44	3.41
query30	0.43	0.24	0.24
query31	2.83	0.41	0.41
query32	3.24	0.49	0.49
query33	2.99	2.98	2.99
query34	16.90	4.40	4.41
query35	4.48	4.47	4.44
query36	0.66	0.48	0.51
query37	0.21	0.17	0.19
query38	0.18	0.17	0.17
query39	0.06	0.06	0.06
query40	0.19	0.17	0.16
query41	0.12	0.08	0.07
query42	0.07	0.07	0.07
query43	0.06	0.06	0.07
Total cold run time: 109.82 s
Total hot run time: 31.81 s

@htyoung htyoung force-pushed the change-list-partition-buckets branch from 116f470 to 766b8a5 Compare August 22, 2024 03:36
@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Aug 22, 2024
Copy link
Contributor

@yujun777 yujun777 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Aug 22, 2024
@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@morningman
Copy link
Contributor

run buildall

morningman pushed a commit that referenced this pull request Aug 23, 2024
)

## Proposed changes
bp #39696

Issue Number: close #39684

```sql
CREATE TABLE `test1` (
    `id1` VARCHAR(255) NULL COMMENT 'id1',
    `id2` VARCHAR(255) NULL COMMENT 'id2',
    `event_time` VARCHAR(255) NULL COMMENT '事件时间',
    `event_date` VARCHAR(255) NULL COMMENT '事件日期',
    `event_ts` VARCHAR(256) NULL COMMENT '事件发生时间戳(毫秒)',
    `dt` VARCHAR(255) NOT NULL COMMENT '日期分区',
    `hr` VARCHAR(255) NOT NULL COMMENT '小时分区'
  ) ENGINE = OLAP DUPLICATE KEY(`id1`) COMMENT 'xxx' PARTITION BY LIST(`dt`, `hr`) (
    PARTITION p2024082021 VALUES IN (("2024-08-20", "21"))
  ) DISTRIBUTED BY HASH(`dt`, `hr`) BUCKETS 2 PROPERTIES (
    "replication_allocation" = "tag.location.default: 1",
    "min_load_replica_num" = "-1",
    "is_being_synced" = "false",
    "storage_medium" = "hdd",
    "storage_format" = "V2",
    "inverted_index_storage_format" = "V1",
    "light_schema_change" = "true",
    "disable_auto_compaction" = "false",
    "enable_single_replica_compaction" = "false",
    "group_commit_interval_ms" = "10000",
    "group_commit_data_bytes" = "134217728"
  );
```

1. 修改前表已有分区的bucket num为2

![image](https://github.com/user-attachments/assets/77efdd0c-f845-41a4-9a31-e454808ffe67)

2. 修改List分区表bucket num(从2 -> 4)

![image](https://github.com/user-attachments/assets/53b19918-2879-4cb3-b2bd-84ba35a7fc59)

3. 修改List分区后,添加的新分区bucket num为4

![image](https://github.com/user-attachments/assets/1f41f73f-d70f-433e-a7b6-8346b7dfcc4e)

Co-authored-by: tongyang.han <tongyang.han@jiduauto.com>
@doris-robot
Copy link

TPC-H: Total hot run time: 37629 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 766b8a5f5442a9fd840706dd3344ac84291ba8b6, data reload: false

------ Round 1 ----------------------------------
q1	17664	4357	4290	4290
q2	2006	184	174	174
q3	11755	969	1159	969
q4	10521	777	753	753
q5	7746	2843	2769	2769
q6	224	139	139	139
q7	980	606	606	606
q8	9332	2064	2079	2064
q9	7262	6537	6558	6537
q10	7006	2196	2150	2150
q11	457	245	241	241
q12	383	222	221	221
q13	17762	3013	3036	3013
q14	281	236	238	236
q15	533	486	494	486
q16	495	397	389	389
q17	972	696	647	647
q18	7312	6885	6811	6811
q19	1378	910	1035	910
q20	680	334	335	334
q21	3871	3016	2865	2865
q22	1120	1040	1025	1025
Total cold run time: 109740 ms
Total hot run time: 37629 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4439	4259	4268	4259
q2	389	281	267	267
q3	2878	2649	2712	2649
q4	1917	1712	1676	1676
q5	5506	5693	5634	5634
q6	233	143	139	139
q7	2308	1777	1831	1777
q8	3299	3423	3423	3423
q9	8828	8825	8830	8825
q10	3545	3353	3369	3353
q11	598	504	514	504
q12	811	663	641	641
q13	13537	3242	3180	3180
q14	320	312	288	288
q15	532	493	503	493
q16	486	447	469	447
q17	1843	1515	1527	1515
q18	8169	7752	7796	7752
q19	1717	1588	1598	1588
q20	2144	1933	1914	1914
q21	5728	5410	5459	5410
q22	1196	1125	1054	1054
Total cold run time: 70423 ms
Total hot run time: 56788 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 191184 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 766b8a5f5442a9fd840706dd3344ac84291ba8b6, data reload: false

query1	1244	891	855	855
query2	6274	1876	1815	1815
query3	10611	4118	3924	3924
query4	59649	23873	23359	23359
query5	5467	493	498	493
query6	402	160	171	160
query7	5767	288	288	288
query8	283	211	203	203
query9	8842	2482	2470	2470
query10	474	267	251	251
query11	18360	15009	15121	15009
query12	165	115	114	114
query13	1595	406	377	377
query14	10748	7553	7468	7468
query15	222	174	179	174
query16	7160	496	530	496
query17	1089	571	583	571
query18	1712	297	294	294
query19	279	150	145	145
query20	118	112	110	110
query21	206	102	102	102
query22	4628	4504	4442	4442
query23	34254	33658	33219	33219
query24	5915	2858	2881	2858
query25	553	395	402	395
query26	689	157	162	157
query27	1768	276	276	276
query28	3819	2082	2054	2054
query29	664	423	418	418
query30	236	168	148	148
query31	934	769	762	762
query32	81	58	58	58
query33	463	297	288	288
query34	891	466	475	466
query35	850	724	709	709
query36	1044	909	912	909
query37	138	81	81	81
query38	4052	3920	3809	3809
query39	1468	1372	1401	1372
query40	194	116	112	112
query41	45	45	44	44
query42	115	99	96	96
query43	483	462	450	450
query44	1076	740	746	740
query45	194	166	162	162
query46	1086	718	737	718
query47	1887	1786	1788	1786
query48	361	290	280	280
query49	758	428	433	428
query50	814	405	420	405
query51	7197	6976	7033	6976
query52	96	87	87	87
query53	250	177	174	174
query54	541	465	473	465
query55	79	74	75	74
query56	269	250	244	244
query57	1207	1042	1049	1042
query58	211	224	240	224
query59	2888	2804	2690	2690
query60	292	269	266	266
query61	103	97	99	97
query62	749	643	645	643
query63	216	181	181	181
query64	3304	1737	1754	1737
query65	3235	3211	3180	3180
query66	663	335	333	333
query67	15415	15450	15146	15146
query68	2954	565	581	565
query69	405	283	294	283
query70	1182	1119	1100	1100
query71	335	274	274	274
query72	2535	2055	2053	2053
query73	703	316	319	316
query74	9082	8710	8701	8701
query75	3370	2651	2733	2651
query76	1412	987	993	987
query77	559	326	314	314
query78	9641	8985	9124	8985
query79	1026	550	538	538
query80	679	493	494	493
query81	464	231	227	227
query82	290	140	142	140
query83	171	146	146	146
query84	250	81	74	74
query85	675	350	282	282
query86	298	303	262	262
query87	4426	4256	4162	4162
query88	3161	2273	2289	2273
query89	375	284	279	279
query90	1953	189	192	189
query91	122	99	98	98
query92	61	53	49	49
query93	1035	529	543	529
query94	683	297	286	286
query95	331	256	260	256
query96	586	264	261	261
query97	3213	3131	3059	3059
query98	222	200	200	200
query99	1478	1289	1273	1273
Total cold run time: 300519 ms
Total hot run time: 191184 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.66 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 766b8a5f5442a9fd840706dd3344ac84291ba8b6, data reload: false

query1	0.05	0.05	0.04
query2	0.08	0.04	0.04
query3	0.23	0.05	0.05
query4	1.67	0.07	0.09
query5	0.50	0.48	0.49
query6	1.13	0.73	0.73
query7	0.02	0.01	0.01
query8	0.06	0.04	0.04
query9	0.54	0.49	0.48
query10	0.54	0.54	0.54
query11	0.15	0.11	0.12
query12	0.15	0.12	0.12
query13	0.61	0.59	0.57
query14	0.75	0.81	0.78
query15	0.88	0.82	0.82
query16	0.38	0.37	0.36
query17	1.02	1.04	0.98
query18	0.21	0.20	0.20
query19	1.90	1.79	1.72
query20	0.01	0.00	0.01
query21	15.40	0.67	0.66
query22	4.51	6.45	2.02
query23	18.28	1.45	1.30
query24	2.15	0.22	0.22
query25	0.15	0.09	0.08
query26	0.26	0.19	0.18
query27	0.08	0.08	0.07
query28	13.22	1.01	1.00
query29	12.61	3.33	3.28
query30	0.24	0.06	0.05
query31	2.88	0.42	0.39
query32	3.23	0.46	0.46
query33	2.96	2.94	3.00
query34	17.19	4.34	4.41
query35	4.42	4.48	4.43
query36	0.66	0.48	0.50
query37	0.18	0.16	0.16
query38	0.15	0.15	0.15
query39	0.05	0.04	0.04
query40	0.15	0.13	0.13
query41	0.10	0.05	0.05
query42	0.06	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 109.86 s
Total hot run time: 30.66 s

Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit e4167e9 into apache:master Aug 23, 2024
dataroaring pushed a commit that referenced this pull request Aug 26, 2024
)

Issue Number: close #39684

Co-authored-by: tongyang.han <tongyang.han@jiduauto.com>
morningman pushed a commit that referenced this pull request Aug 28, 2024
…ket num (#40033)

cherry pick from #39696
Co-authored-by: tongyang.han <tongyang.han@jiduauto.com>
morningman pushed a commit that referenced this pull request Aug 30, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] can't change the default bucket num for List partition table

6 participants