Skip to content

[chore](fe) Returns dropped tables in GetMeta request #38019

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
w41ter merged 1 commit into from
Jul 18, 2024
Merged

[chore](fe) Returns dropped tables in GetMeta request #38019

w41ter merged 1 commit into from
Jul 18, 2024

Conversation

@w41ter
Copy link
Contributor

@w41ter w41ter commented Jul 17, 2024

The CCR syncer needs to know the distribution of tables, partitions, indexes, and replicas when synchronizing binlogs. If a table is deleted before the binlog synchronization is complete, the CCR syncer cannot continue synchronizing. This PR will record the deleted tables and include them in the get meta response, allowing the CCR syncer to filter out the binlogs that belong to these tables.

The CCR syncer part PR is selectdb/ccr-syncer#123.

@w41ter
[chore](fe) Returns dropped tables in GetMeta request
ff89966 
The CCR syncer needs to know the distribution of tables, partitions,
indexes, and replicas when synchronizing binlogs. If a table is
deleted before the binlog synchronization is complete, the CCR syncer
cannot continue synchronizing. This PR will record the deleted
tables and include them in the get meta response, allowing the CCR
syncer to filter out the binlogs that belong to these tables.

The CCR syncer part PR is selectdb/ccr-syncer#123.
@doris-robot
Copy link

doris-robot commented Jul 17, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@w41ter
Copy link
Contributor Author

w41ter commented Jul 17, 2024

run buildall

@github-actions
Copy link
Contributor

github-actions bot commented Jul 17, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

doris-robot commented Jul 17, 2024

TPC-H: Total hot run time: 41723 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ff89966af5a17bc56e150b9d203a2fed5ce7f137, data reload: false

------ Round 1 ----------------------------------
q1	18157	4619	4565	4565
q2	2026	198	202	198
q3	10517	1295	1240	1240
q4	10233	884	815	815
q5	7568	2842	2791	2791
q6	240	139	141	139
q7	965	623	631	623
q8	9225	2286	2314	2286
q9	9036	6856	6855	6855
q10	8887	3926	3892	3892
q11	461	238	241	238
q12	438	223	236	223
q13	17767	3015	3041	3015
q14	290	244	247	244
q15	546	471	501	471
q16	510	399	390	390
q17	1043	693	661	661
q18	8273	7598	7500	7500
q19	7295	1521	1481	1481
q20	700	324	327	324
q21	5170	3467	4138	3467
q22	364	316	305	305
Total cold run time: 119711 ms
Total hot run time: 41723 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4643	4458	4579	4458
q2	422	280	274	274
q3	3322	3020	3060	3020
q4	2138	1714	1785	1714
q5	5660	5630	5754	5630
q6	243	144	137	137
q7	2345	1858	1898	1858
q8	3553	3737	3751	3737
q9	8927	8906	8937	8906
q10	4197	4041	4053	4041
q11	625	534	496	496
q12	879	671	646	646
q13	17358	3228	3146	3146
q14	321	301	295	295
q15	552	501	513	501
q16	517	472	443	443
q17	2011	1641	1584	1584
q18	8212	8029	7906	7906
q19	1916	1794	1791	1791
q20	2227	1936	1902	1902
q21	5265	4960	5139	4960
q22	625	561	516	516
Total cold run time: 75958 ms
Total hot run time: 57961 ms

@doris-robot
Copy link

doris-robot commented Jul 17, 2024

TPC-DS: Total hot run time: 175157 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ff89966af5a17bc56e150b9d203a2fed5ce7f137, data reload: false

query1	923	375	366	366
query2	6815	1831	1887	1831
query3	6634	218	221	218
query4	28355	17640	17601	17601
query5	3813	483	478	478
query6	271	178	158	158
query7	4582	294	295	294
query8	247	194	198	194
query9	8423	2390	2357	2357
query10	431	267	291	267
query11	10563	10077	10211	10077
query12	111	87	81	81
query13	1637	363	363	363
query14	10104	7895	8110	7895
query15	229	165	170	165
query16	7168	327	322	322
query17	1409	547	519	519
query18	1752	288	294	288
query19	192	148	147	147
query20	88	80	79	79
query21	205	136	125	125
query22	4366	4105	4102	4102
query23	34218	33752	33645	33645
query24	9316	2959	2936	2936
query25	606	393	407	393
query26	705	148	148	148
query27	2264	284	282	282
query28	5925	2072	2063	2063
query29	898	644	634	634
query30	266	162	153	153
query31	983	784	748	748
query32	97	56	58	56
query33	628	319	372	319
query34	894	500	500	500
query35	684	610	602	602
query36	1140	1008	968	968
query37	146	82	86	82
query38	2999	2905	2877	2877
query39	899	861	856	856
query40	206	122	121	121
query41	46	45	44	44
query42	114	102	96	96
query43	500	461	482	461
query44	1061	720	712	712
query45	195	165	158	158
query46	1087	716	721	716
query47	1851	1824	1798	1798
query48	372	285	284	284
query49	815	410	446	410
query50	777	393	390	390
query51	6942	6851	6672	6672
query52	107	107	94	94
query53	355	292	286	286
query54	748	445	445	445
query55	77	78	73	73
query56	281	259	275	259
query57	1126	1084	1076	1076
query58	255	253	245	245
query59	2863	2619	2573	2573
query60	302	269	278	269
query61	97	94	98	94
query62	784	636	656	636
query63	327	301	290	290
query64	9188	2296	7567	2296
query65	3144	3130	3106	3106
query66	765	332	321	321
query67	15631	14987	15066	14987
query68	4504	530	531	530
query69	473	317	335	317
query70	1152	1193	1158	1158
query71	368	270	281	270
query72	6990	5557	5692	5557
query73	723	332	324	324
query74	6135	5817	5715	5715
query75	3374	2691	2660	2660
query76	2134	952	940	940
query77	436	305	317	305
query78	9659	9082	8985	8985
query79	2688	532	517	517
query80	2702	480	475	475
query81	609	227	219	219
query82	832	132	138	132
query83	306	179	165	165
query84	283	91	84	84
query85	2025	318	304	304
query86	475	307	323	307
query87	3302	3135	3153	3135
query88	3847	2388	2363	2363
query89	482	393	391	391
query90	1729	197	192	192
query91	141	111	114	111
query92	64	52	54	52
query93	3570	516	511	511
query94	1155	225	226	225
query95	417	323	331	323
query96	615	280	272	272
query97	3189	3073	3081	3073
query98	240	203	196	196
query99	1674	1271	1248	1248
Total cold run time: 277295 ms
Total hot run time: 175157 ms

@doris-robot
Copy link

doris-robot commented Jul 17, 2024

ClickBench: Total hot run time: 30.91 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ff89966af5a17bc56e150b9d203a2fed5ce7f137, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.04	0.04
query3	0.23	0.05	0.06
query4	1.69	0.08	0.08
query5	0.51	0.50	0.49
query6	1.14	0.72	0.72
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.55	0.49	0.49
query10	0.56	0.52	0.55
query11	0.15	0.11	0.12
query12	0.16	0.13	0.13
query13	0.60	0.58	0.58
query14	0.77	0.79	0.79
query15	0.86	0.81	0.82
query16	0.36	0.37	0.37
query17	1.04	1.05	1.02
query18	0.23	0.22	0.21
query19	1.92	1.85	1.84
query20	0.01	0.01	0.01
query21	15.45	0.76	0.66
query22	4.33	6.58	1.89
query23	18.28	1.38	1.25
query24	2.06	0.25	0.23
query25	0.15	0.09	0.08
query26	0.29	0.21	0.22
query27	0.45	0.23	0.23
query28	13.25	1.03	1.01
query29	12.66	3.31	3.29
query30	0.25	0.06	0.06
query31	2.86	0.40	0.39
query32	3.27	0.48	0.46
query33	2.87	2.94	2.89
query34	17.12	4.36	4.37
query35	4.41	4.49	4.51
query36	0.65	0.48	0.48
query37	0.18	0.15	0.16
query38	0.15	0.15	0.15
query39	0.04	0.04	0.03
query40	0.15	0.11	0.12
query41	0.10	0.05	0.05
query42	0.06	0.06	0.06
query43	0.05	0.04	0.04
Total cold run time: 110.05 s
Total hot run time: 30.91 s

dataroaring
dataroaring approved these changes Jul 17, 2024
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 17, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Jul 17, 2024

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Jul 17, 2024

PR approved by anyone and no changes requested.

@w41ter w41ter merged commit ddcbb1c into master Jul 18, 2024
@w41ter w41ter deleted the filter_dropped_tables branch July 18, 2024 02:01
dataroaring pushed a commit that referenced this pull request Jul 19, 2024
@w41ter @dataroaring
[chore](fe) Returns dropped tables in GetMeta request (#38019)
67aa261 
The CCR syncer needs to know the distribution of tables, partitions,
indexes, and replicas when synchronizing binlogs. If a table is deleted
before the binlog synchronization is complete, the CCR syncer cannot
continue synchronizing. This PR will record the deleted tables and
include them in the get meta response, allowing the CCR syncer to filter
out the binlogs that belong to these tables.

The CCR syncer part PR is selectdb/ccr-syncer#123.
w41ter added a commit to w41ter/incubator-doris that referenced this pull request Jul 30, 2024
@w41ter
[chore](fe) Returns dropped tables in GetMeta request
6048814 
Cherry-pick apache#38019
@w41ter w41ter mentioned this pull request Jul 30, 2024
Merged
w41ter added a commit to w41ter/incubator-doris that referenced this pull request Jul 30, 2024
@w41ter
[chore](fe) Returns dropped tables in GetMeta request
a29afde 
Cherry-pick apache#38019
@w41ter w41ter mentioned this pull request Jul 30, 2024
Merged
w41ter added a commit that referenced this pull request Jul 31, 2024
@w41ter
[chore](fe) Returns dropped tables in GetMeta request (#38542)
ce7c979 
Cherry-pick #38019
w41ter added a commit that referenced this pull request Jul 31, 2024
@w41ter
[chore](fe) Returns dropped tables in GetMeta request (#38541)
182bf4d 
Cherry-pick #38019
mongo360 pushed a commit to mongo360/doris that referenced this pull request Dec 11, 2024
@w41ter
[chore](fe) Returns dropped tables in GetMeta request (apache#38542)
37d7d96 
Cherry-pick apache#38019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@CalvinKirs CalvinKirs CalvinKirs approved these changes

@dataroaring dataroaring dataroaring approved these changes

@morningman morningman Awaiting requested review from morningman morningman is a code owner

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.0.14-merged dev/2.1.6-merged dev/3.0.1-merged meta-change reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@w41ter @doris-robot @CalvinKirs @dataroaring