Skip to content

[improvement](binlog)Support inverted index in CCR (#31743) #36203

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
xiaokang merged 3 commits into from
Jun 16, 2024
Merged

[improvement](binlog)Support inverted index in CCR (#31743) #36203

xiaokang merged 3 commits into from
Jun 16, 2024

Conversation

@qidaye
Copy link
Contributor

@qidaye qidaye commented Jun 12, 2024

pick from master #31743

@doris-robot
Copy link

doris-robot commented Jun 12, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@qidaye
Copy link
Contributor Author

qidaye commented Jun 12, 2024

run buildall

1 similar comment
@qidaye
Copy link
Contributor Author

qidaye commented Jun 12, 2024

run buildall

@github-actions
Copy link
Contributor

github-actions bot commented Jun 12, 2024

clang-tidy review says "All clean, LGTM! 👍"

1 similar comment
@github-actions
Copy link
Contributor

github-actions bot commented Jun 12, 2024

clang-tidy review says "All clean, LGTM! 👍"

@qidaye
Copy link
Contributor Author

qidaye commented Jun 12, 2024

run buildall

@github-actions
Copy link
Contributor

github-actions bot commented Jun 12, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

doris-robot commented Jun 12, 2024

TPC-H: Total hot run time: 50032 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a0388701ead424e2cd6586191fa573de5313c176, data reload: false

------ Round 1 ----------------------------------
q1	17712	4475	4359	4359
q2	2047	157	145	145
q3	10462	1886	1949	1886
q4	10380	1282	1328	1282
q5	8634	3942	3903	3903
q6	235	124	126	124
q7	2018	1600	1609	1600
q8	9339	2766	2795	2766
q9	10705	10297	10263	10263
q10	8698	3497	3527	3497
q11	427	249	253	249
q12	474	301	311	301
q13	18340	3979	4014	3979
q14	366	334	329	329
q15	508	461	453	453
q16	692	580	571	571
q17	1160	962	997	962
q18	7220	6941	6964	6941
q19	1784	1649	1672	1649
q20	545	306	290	290
q21	4429	4153	4050	4050
q22	525	446	433	433
Total cold run time: 116700 ms
Total hot run time: 50032 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4328	4324	4352	4324
q2	326	228	226	226
q3	4192	4170	4200	4170
q4	2747	2750	2743	2743
q5	7228	7169	7121	7121
q6	236	121	123	121
q7	3277	2896	2889	2889
q8	4372	4462	4522	4462
q9	16820	17628	16887	16887
q10	4282	4258	4223	4223
q11	806	703	745	703
q12	1029	863	876	863
q13	6867	3754	3749	3749
q14	461	432	414	414
q15	516	463	458	458
q16	737	709	688	688
q17	3833	3896	3905	3896
q18	8733	8857	8863	8857
q19	1701	1681	1661	1661
q20	2447	2107	2099	2099
q21	8538	8572	8451	8451
q22	1051	998	1003	998
Total cold run time: 84527 ms
Total hot run time: 80003 ms

@doris-robot
Copy link

doris-robot commented Jun 12, 2024

TPC-DS: Total hot run time: 204230 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a0388701ead424e2cd6586191fa573de5313c176, data reload: false

query1	948	430	377	377
query2	6524	2707	2716	2707
query3	6935	209	202	202
query4	21256	18104	18078	18078
query5	19725	6507	6498	6498
query6	282	223	231	223
query7	4156	304	315	304
query8	482	413	437	413
query9	3057	2661	2593	2593
query10	401	307	310	307
query11	11398	10806	10834	10806
query12	123	78	74	74
query13	5595	677	668	668
query14	17978	13765	13403	13403
query15	355	249	251	249
query16	6479	281	265	265
query17	1701	1446	893	893
query18	2305	411	405	405
query19	210	146	149	146
query20	83	77	78	77
query21	189	94	92	92
query22	5313	5156	5143	5143
query23	32465	31964	32022	31964
query24	6884	6535	6514	6514
query25	532	441	422	422
query26	512	164	163	163
query27	1766	296	291	291
query28	6164	2331	2315	2315
query29	2919	2742	2819	2742
query30	246	164	164	164
query31	934	728	726	726
query32	71	63	63	63
query33	401	268	261	261
query34	859	468	492	468
query35	1110	948	874	874
query36	1257	1367	1323	1323
query37	90	60	64	60
query38	3038	2927	2900	2900
query39	1384	1347	1315	1315
query40	209	93	99	93
query41	46	44	45	44
query42	82	87	87	87
query43	766	632	654	632
query44	1118	713	723	713
query45	249	233	230	230
query46	1222	971	978	971
query47	2039	1772	1708	1708
query48	1021	711	721	711
query49	628	369	374	369
query50	851	579	596	579
query51	4717	4668	4683	4668
query52	88	82	84	82
query53	458	335	331	331
query54	2671	2444	2443	2443
query55	96	84	83	83
query56	228	212	211	211
query57	1147	1219	1126	1126
query58	216	203	205	203
query59	4000	4255	4052	4052
query60	215	206	218	206
query61	95	95	95	95
query62	910	474	494	474
query63	488	342	342	342
query64	2505	1554	1497	1497
query65	3622	3528	3541	3528
query66	778	381	376	376
query67	16534	15390	16343	15390
query68	8820	691	653	653
query69	558	365	356	356
query70	1644	1379	1527	1379
query71	400	302	314	302
query72	6743	3507	3460	3460
query73	741	319	316	316
query74	6337	5876	5944	5876
query75	5285	3729	3554	3554
query76	5536	1176	1168	1168
query77	954	253	255	253
query78	12346	11948	11485	11485
query79	10031	664	650	650
query80	1596	405	403	403
query81	489	233	228	228
query82	1578	103	105	103
query83	184	137	135	135
query84	256	70	69	69
query85	879	315	315	315
query86	332	295	312	295
query87	3210	3026	3069	3026
query88	5106	2336	2343	2336
query89	482	300	306	300
query90	1920	209	206	206
query91	173	139	138	138
query92	57	51	54	51
query93	6462	574	542	542
query94	716	214	209	209
query95	1118	1062	1056	1056
query96	643	323	325	323
query97	6463	6374	6337	6337
query98	189	171	166	166
query99	3005	851	808	808
Total cold run time: 317379 ms
Total hot run time: 204230 ms

@doris-robot
Copy link

doris-robot commented Jun 12, 2024

ClickBench: Total hot run time: 30.89 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a0388701ead424e2cd6586191fa573de5313c176, data reload: false

query1	0.02	0.02	0.02
query2	0.07	0.02	0.02
query3	0.25	0.05	0.05
query4	1.79	0.07	0.08
query5	0.53	0.52	0.52
query6	1.23	0.60	0.60
query7	0.01	0.01	0.01
query8	0.03	0.02	0.02
query9	0.54	0.49	0.48
query10	0.54	0.55	0.53
query11	0.11	0.08	0.08
query12	0.12	0.09	0.09
query13	0.61	0.62	0.61
query14	0.80	0.79	0.79
query15	0.78	0.76	0.76
query16	0.36	0.35	0.37
query17	1.06	1.01	0.99
query18	0.24	0.24	0.26
query19	1.98	1.87	1.87
query20	0.02	0.01	0.01
query21	15.48	0.56	0.57
query22	2.14	2.26	1.82
query23	17.00	0.96	0.96
query24	6.08	1.07	1.01
query25	0.38	0.09	0.05
query26	0.66	0.17	0.16
query27	0.05	0.04	0.04
query28	6.95	0.74	0.73
query29	12.74	2.19	2.06
query30	0.52	0.53	0.54
query31	2.80	0.39	0.38
query32	3.39	0.49	0.49
query33	3.03	3.15	3.11
query34	15.25	4.81	4.81
query35	4.85	4.83	4.82
query36	1.05	1.00	1.04
query37	0.06	0.05	0.04
query38	0.03	0.02	0.02
query39	0.02	0.02	0.01
query40	0.16	0.14	0.14
query41	0.06	0.02	0.01
query42	0.02	0.02	0.01
query43	0.02	0.02	0.02
Total cold run time: 103.83 s
Total hot run time: 30.89 s

@doris-robot
Copy link

doris-robot commented Jun 12, 2024

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit a0388701ead424e2cd6586191fa573de5313c176 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.4 seconds inserted 10000000 Rows, about 467K ops/s

@qidaye
Copy link
Contributor Author

qidaye commented Jun 13, 2024

run beut

@qidaye
Copy link
Contributor Author

qidaye commented Jun 13, 2024

run buildall

@github-actions
Copy link
Contributor

github-actions bot commented Jun 13, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

doris-robot commented Jun 13, 2024

TPC-H: Total hot run time: 49723 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d8c22d5dcb81b6d996a57af0da97c494ff9e4872, data reload: false

------ Round 1 ----------------------------------
q1	17663	4368	4355	4355
q2	2059	153	144	144
q3	10298	1898	1965	1898
q4	10345	1206	1329	1206
q5	8750	3924	3894	3894
q6	234	142	124	124
q7	2025	1577	1613	1577
q8	9300	2732	2718	2718
q9	10721	10161	10199	10161
q10	8601	3527	3522	3522
q11	409	252	252	252
q12	470	308	309	308
q13	18332	3961	4029	3961
q14	361	330	322	322
q15	517	467	451	451
q16	691	577	583	577
q17	1139	947	945	945
q18	7323	6906	6860	6860
q19	1762	1634	1599	1599
q20	517	307	283	283
q21	4430	4149	4123	4123
q22	532	443	447	443
Total cold run time: 116479 ms
Total hot run time: 49723 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4372	4319	4384	4319
q2	328	234	227	227
q3	4153	4153	4161	4153
q4	2748	2737	2742	2737
q5	7185	7078	7083	7078
q6	236	117	122	117
q7	3217	2916	2857	2857
q8	4367	4438	4445	4438
q9	16905	16823	16727	16727
q10	4240	4287	4264	4264
q11	744	682	664	664
q12	1010	869	844	844
q13	7180	3739	3764	3739
q14	440	440	427	427
q15	511	453	465	453
q16	747	689	690	689
q17	3921	3907	3913	3907
q18	8794	8689	8774	8689
q19	1738	1729	1655	1655
q20	2342	2160	2117	2117
q21	8424	8417	8545	8417
q22	1032	971	985	971
Total cold run time: 84634 ms
Total hot run time: 79489 ms

@doris-robot
Copy link

doris-robot commented Jun 13, 2024

TPC-DS: Total hot run time: 203721 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d8c22d5dcb81b6d996a57af0da97c494ff9e4872, data reload: false

query1	938	416	386	386
query2	6528	2915	2637	2637
query3	6925	206	198	198
query4	20238	18138	17992	17992
query5	19726	6465	6518	6465
query6	302	224	231	224
query7	4151	311	306	306
query8	448	431	413	413
query9	3126	2703	2615	2615
query10	418	299	305	299
query11	11384	10728	10777	10728
query12	126	74	74	74
query13	5590	691	678	678
query14	17767	13434	14072	13434
query15	367	236	255	236
query16	6474	288	264	264
query17	1736	1482	874	874
query18	2289	420	420	420
query19	218	149	145	145
query20	80	80	80	80
query21	194	93	98	93
query22	5057	5083	4985	4985
query23	32662	31979	32076	31979
query24	6970	6519	6546	6519
query25	512	428	436	428
query26	527	159	168	159
query27	1893	301	304	301
query28	6161	2366	2313	2313
query29	3044	2721	2886	2721
query30	245	163	169	163
query31	902	738	760	738
query32	71	63	60	60
query33	411	263	263	263
query34	842	499	482	482
query35	1130	946	940	940
query36	1356	1092	1220	1092
query37	95	61	58	58
query38	3067	2936	2917	2917
query39	1356	1357	1330	1330
query40	221	98	98	98
query41	51	45	44	44
query42	81	79	86	79
query43	762	721	689	689
query44	1113	713	719	713
query45	245	234	235	234
query46	1234	972	972	972
query47	1941	1825	1988	1825
query48	1009	720	709	709
query49	617	376	379	376
query50	869	616	640	616
query51	4819	4640	4668	4640
query52	95	89	95	89
query53	442	334	323	323
query54	2630	2439	2444	2439
query55	92	76	83	76
query56	237	225	200	200
query57	1118	1052	1199	1052
query58	228	198	209	198
query59	4092	3927	3916	3916
query60	220	201	219	201
query61	97	93	96	93
query62	711	485	468	468
query63	487	347	337	337
query64	2609	1524	1492	1492
query65	3666	3568	3556	3556
query66	773	379	382	379
query67	16122	15523	15176	15176
query68	10628	665	644	644
query69	581	367	355	355
query70	2059	1665	1304	1304
query71	430	323	305	305
query72	6573	3476	3512	3476
query73	740	324	317	317
query74	6318	5827	5874	5827
query75	5427	3705	3782	3705
query76	6753	1206	1151	1151
query77	1190	258	255	255
query78	12600	12143	11679	11679
query79	11326	668	638	638
query80	1130	408	390	390
query81	493	237	234	234
query82	1100	99	98	98
query83	167	135	133	133
query84	262	72	73	72
query85	872	318	313	313
query86	337	316	340	316
query87	3210	3025	3036	3025
query88	5088	2328	2308	2308
query89	428	301	280	280
query90	2099	215	209	209
query91	174	142	140	140
query92	64	54	56	54
query93	5932	569	550	550
query94	766	215	206	206
query95	1099	1062	1060	1060
query96	668	325	335	325
query97	6519	6334	6352	6334
query98	188	173	181	173
query99	3137	846	855	846
Total cold run time: 320265 ms
Total hot run time: 203721 ms

@doris-robot
Copy link

doris-robot commented Jun 13, 2024

TeamCity be ut coverage result:
Function Coverage: 37.79% (8094/21416)
Line Coverage: 29.45% (66123/224550)
Region Coverage: 28.92% (34083/117839)
Branch Coverage: 24.79% (17492/70562)
Coverage Report: http://coverage.selectdb-in.cc/coverage/d8c22d5dcb81b6d996a57af0da97c494ff9e4872_d8c22d5dcb81b6d996a57af0da97c494ff9e4872/report/index.html

@doris-robot
Copy link

doris-robot commented Jun 13, 2024

ClickBench: Total hot run time: 30.74 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d8c22d5dcb81b6d996a57af0da97c494ff9e4872, data reload: false

query1	0.03	0.03	0.02
query2	0.07	0.03	0.03
query3	0.25	0.05	0.05
query4	1.78	0.07	0.08
query5	0.54	0.53	0.51
query6	1.23	0.60	0.60
query7	0.01	0.01	0.01
query8	0.03	0.02	0.03
query9	0.52	0.49	0.48
query10	0.54	0.55	0.55
query11	0.12	0.09	0.08
query12	0.12	0.09	0.09
query13	0.63	0.61	0.60
query14	0.79	0.80	0.78
query15	0.78	0.76	0.78
query16	0.39	0.39	0.37
query17	1.00	1.04	1.01
query18	0.23	0.25	0.24
query19	1.89	1.88	1.87
query20	0.02	0.01	0.01
query21	15.49	0.54	0.57
query22	2.31	2.16	1.85
query23	17.12	1.13	0.88
query24	4.46	0.80	1.32
query25	0.31	0.12	0.05
query26	0.56	0.16	0.15
query27	0.03	0.04	0.03
query28	8.51	0.73	0.71
query29	12.62	2.40	2.17
query30	0.62	0.51	0.52
query31	2.82	0.38	0.38
query32	3.39	0.50	0.49
query33	3.07	3.09	3.09
query34	15.25	4.81	4.78
query35	4.85	4.87	4.87
query36	1.09	1.04	1.03
query37	0.06	0.05	0.05
query38	0.03	0.02	0.02
query39	0.02	0.02	0.01
query40	0.16	0.15	0.14
query41	0.07	0.01	0.02
query42	0.02	0.01	0.02
query43	0.03	0.02	0.02
Total cold run time: 103.86 s
Total hot run time: 30.74 s

@doris-robot
Copy link

doris-robot commented Jun 13, 2024

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit d8c22d5dcb81b6d996a57af0da97c494ff9e4872 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.6 seconds inserted 10000000 Rows, about 462K ops/s

@xiaokang xiaokang merged commit 09b56d5 into apache:branch-2.0 Jun 16, 2024
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
@qidaye
[improvement](binlog)Support inverted index in CCR (apache#31743) (ap…
d8cd2c9 
…ache#36203)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@qidaye @doris-robot @xiaokang