Skip to content

Conversation

@w41ter
Copy link
Contributor

@w41ter w41ter commented Jul 18, 2024

Cherry-pick #37196, #37326.

The CCR syncer needs to know the distribution of tables, partitions,
indexes, and replicas when synchronizing binlogs. If a partition is
deleted before the binlog synchronization is complete, the CCR syncer
cannot continue synchronizing. This PR will record the deleted
partitions and include them in the get meta response, allowing the CCR
syncer to filter out the binlogs that belong to these partitions.

The CCR syncer part PR is selectdb/ccr-syncer#117.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@w41ter
Copy link
Contributor Author

w41ter commented Jul 18, 2024

run buildall

@github-actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

w41ter added 2 commits July 18, 2024 08:31
Cherry-pick apache#37196.

The CCR syncer needs to know the distribution of tables, partitions,
indexes, and replicas when synchronizing binlogs. If a partition is
deleted before the binlog synchronization is complete, the CCR syncer
cannot continue synchronizing. This PR will record the deleted
partitions and include them in the get meta response, allowing the CCR
syncer to filter out the binlogs that belong to these partitions.

The CCR syncer part PR is selectdb/ccr-syncer#117.
@w41ter w41ter force-pushed the pick/37196_to_2.0 branch from 7e1f530 to 126fbb5 Compare July 18, 2024 08:31
@w41ter
Copy link
Contributor Author

w41ter commented Jul 18, 2024

run buildall

@github-actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 49807 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 126fbb527b000e1c80a060b3b83be2dabd9d5007, data reload: false

------ Round 1 ----------------------------------
q1	17706	4415	4364	4364
q2	2105	163	155	155
q3	10321	1864	1940	1864
q4	10114	1291	1318	1291
q5	8501	3978	3940	3940
q6	235	147	142	142
q7	2014	1602	1593	1593
q8	9494	2756	2729	2729
q9	13574	10356	10192	10192
q10	8613	3513	3507	3507
q11	400	252	251	251
q12	469	302	305	302
q13	18348	3939	4012	3939
q14	345	327	321	321
q15	511	451	459	451
q16	658	580	570	570
q17	1144	983	910	910
q18	7237	6918	6821	6821
q19	1810	1690	1646	1646
q20	511	309	296	296
q21	4441	4082	4115	4082
q22	534	441	453	441
Total cold run time: 119085 ms
Total hot run time: 49807 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4294	4330	4315	4315
q2	324	219	223	219
q3	4169	4115	4130	4115
q4	2733	2741	2748	2741
q5	7156	7064	7064	7064
q6	240	119	122	119
q7	3237	2830	2858	2830
q8	4352	4441	4470	4441
q9	16894	16819	16577	16577
q10	4257	4252	4269	4252
q11	718	697	673	673
q12	1033	840	875	840
q13	7228	3759	3732	3732
q14	451	437	439	437
q15	499	463	458	458
q16	732	666	677	666
q17	3825	3755	3823	3755
q18	8827	8722	8890	8722
q19	1731	1756	1673	1673
q20	2375	2112	2115	2112
q21	8544	8403	8376	8376
q22	1036	957	983	957
Total cold run time: 84655 ms
Total hot run time: 79074 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 203519 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 126fbb527b000e1c80a060b3b83be2dabd9d5007, data reload: false

query1	928	420	376	376
query2	6541	2923	2930	2923
query3	6919	209	205	205
query4	20996	17944	17907	17907
query5	19741	6499	6496	6496
query6	301	217	223	217
query7	4158	299	316	299
query8	409	444	387	387
query9	3102	2668	2590	2590
query10	423	290	296	290
query11	11367	10803	10724	10724
query12	129	80	77	77
query13	5616	709	701	701
query14	17726	13806	13159	13159
query15	366	247	246	246
query16	6464	287	261	261
query17	1706	1457	865	865
query18	2313	411	409	409
query19	205	154	168	154
query20	80	79	80	79
query21	193	97	92	92
query22	5334	5142	5096	5096
query23	32618	31731	31945	31731
query24	6944	6469	6493	6469
query25	506	439	431	431
query26	527	166	160	160
query27	1889	293	290	290
query28	6086	2401	2340	2340
query29	2887	2696	2775	2696
query30	248	174	172	172
query31	919	750	729	729
query32	71	65	63	63
query33	407	264	263	263
query34	849	478	472	472
query35	1123	941	883	883
query36	1479	1191	1073	1073
query37	90	63	60	60
query38	3073	2900	2924	2900
query39	1380	1315	1328	1315
query40	206	97	94	94
query41	56	44	44	44
query42	85	81	88	81
query43	735	667	758	667
query44	1133	717	724	717
query45	250	237	234	234
query46	1254	948	981	948
query47	1782	1692	1723	1692
query48	1001	694	706	694
query49	636	373	366	366
query50	847	599	650	599
query51	4843	4621	4668	4621
query52	93	76	93	76
query53	445	317	321	317
query54	2647	2418	2462	2418
query55	87	87	84	84
query56	247	229	209	209
query57	1206	1075	1190	1075
query58	219	209	203	203
query59	4348	4085	4199	4085
query60	216	219	210	210
query61	97	95	118	95
query62	861	458	504	458
query63	484	330	335	330
query64	2573	1546	1449	1449
query65	3620	3565	3567	3565
query66	788	377	392	377
query67	16014	15808	15304	15304
query68	9910	677	646	646
query69	577	350	356	350
query70	1678	1478	1470	1470
query71	419	314	337	314
query72	6553	3460	3473	3460
query73	737	314	318	314
query74	6283	5879	5893	5879
query75	5356	3672	3666	3666
query76	6345	1147	1215	1147
query77	1057	259	258	258
query78	12510	11726	12539	11726
query79	10971	639	629	629
query80	1233	400	402	400
query81	499	233	234	233
query82	771	97	101	97
query83	172	130	132	130
query84	257	71	73	71
query85	901	344	332	332
query86	339	292	331	292
query87	3311	2999	3060	2999
query88	4540	2332	2320	2320
query89	369	308	307	307
query90	1979	211	210	210
query91	174	153	140	140
query92	62	54	58	54
query93	4139	551	611	551
query94	741	214	202	202
query95	1136	1050	1045	1045
query96	642	329	324	324
query97	6472	6286	6403	6286
query98	190	175	178	175
query99	3045	1083	880	880
Total cold run time: 316254 ms
Total hot run time: 203519 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.96 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 126fbb527b000e1c80a060b3b83be2dabd9d5007, data reload: false

query1	0.02	0.02	0.02
query2	0.07	0.03	0.02
query3	0.25	0.05	0.04
query4	1.77	0.06	0.06
query5	0.57	0.53	0.53
query6	1.23	0.62	0.61
query7	0.02	0.01	0.01
query8	0.03	0.03	0.02
query9	0.53	0.50	0.48
query10	0.54	0.54	0.54
query11	0.12	0.09	0.09
query12	0.11	0.09	0.09
query13	0.64	0.63	0.60
query14	0.79	0.79	0.77
query15	0.79	0.77	0.75
query16	0.38	0.38	0.36
query17	1.02	1.01	1.02
query18	0.23	0.24	0.28
query19	1.86	1.83	1.80
query20	0.02	0.01	0.01
query21	15.51	0.55	0.56
query22	2.00	2.26	1.85
query23	17.18	1.12	0.94
query24	4.88	3.72	0.87
query25	0.36	0.13	0.05
query26	0.78	0.17	0.16
query27	0.05	0.04	0.04
query28	5.76	0.77	0.74
query29	12.62	2.37	2.34
query30	0.60	0.55	0.55
query31	2.81	0.40	0.37
query32	3.36	0.49	0.49
query33	3.07	3.08	3.08
query34	15.29	4.80	4.81
query35	4.85	4.85	4.83
query36	1.04	1.02	1.00
query37	0.06	0.04	0.04
query38	0.03	0.02	0.02
query39	0.02	0.01	0.02
query40	0.16	0.14	0.14
query41	0.06	0.01	0.02
query42	0.02	0.02	0.01
query43	0.02	0.02	0.02
Total cold run time: 101.52 s
Total hot run time: 30.96 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 126fbb527b000e1c80a060b3b83be2dabd9d5007 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.3 seconds inserted 10000000 Rows, about 469K ops/s

@dataroaring dataroaring merged commit 15230b9 into apache:branch-2.0 Jul 18, 2024
@w41ter w41ter deleted the pick/37196_to_2.0 branch July 19, 2024 03:08
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
Cherry-pick apache#37196, apache#37326.

The CCR syncer needs to know the distribution of tables, partitions,
indexes, and replicas when synchronizing binlogs. If a partition is
deleted before the binlog synchronization is complete, the CCR syncer
cannot continue synchronizing. This PR will record the deleted
partitions and include them in the get meta response, allowing the CCR
syncer to filter out the binlogs that belong to these partitions.

The CCR syncer part PR is selectdb/ccr-syncer#117.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants