Skip to content

[opt](mow) opt mow lookup with sequence column (#38287) #38407

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
cambyzju merged 1 commit into from
Jul 30, 2024
Merged

[opt](mow) opt mow lookup with sequence column (#38287) #38407

cambyzju merged 1 commit into from
Jul 30, 2024

Conversation

@cambyzju
Copy link
Contributor

@cambyzju cambyzju commented Jul 26, 2024

pick #38287 to branch-2.0

@cambyzju
[opt](mow) opt mow lookup with sequence column (apache#38287)
8b93924 
Compare primary key without sequence column is expensive,
if full key is exact_match, the primary key without sequence column
should also the same,
do not need to compare again.
@doris-robot
Copy link

doris-robot commented Jul 26, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@cambyzju
Copy link
Contributor Author

cambyzju commented Jul 26, 2024

run buildall

@github-actions
Copy link
Contributor

github-actions bot commented Jul 26, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

doris-robot commented Jul 26, 2024

TPC-H: Total hot run time: 49785 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8b9392479b7c1e5e444dc6429bc93b563f37f449, data reload: false

------ Round 1 ----------------------------------
q1	17664	4357	4376	4357
q2	2030	159	152	152
q3	10419	1895	1932	1895
q4	10309	1242	1332	1242
q5	8478	3906	3894	3894
q6	237	128	126	126
q7	2019	1610	1593	1593
q8	9348	2733	2694	2694
q9	10560	10400	10179	10179
q10	8628	3514	3529	3514
q11	428	258	253	253
q12	473	298	303	298
q13	18380	3961	3994	3961
q14	365	329	334	329
q15	517	468	461	461
q16	706	572	582	572
q17	1124	968	949	949
q18	7340	6927	6817	6817
q19	1810	1682	1632	1632
q20	538	311	290	290
q21	4466	4183	4126	4126
q22	521	451	458	451
Total cold run time: 116360 ms
Total hot run time: 49785 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4325	4308	4294	4294
q2	328	231	232	231
q3	4170	4157	4173	4157
q4	2732	2749	2727	2727
q5	7157	6977	7075	6977
q6	238	116	118	116
q7	3246	2859	2846	2846
q8	4351	4489	4471	4471
q9	16873	16719	16736	16719
q10	4242	4252	4283	4252
q11	748	658	673	658
q12	1036	863	855	855
q13	7309	3742	3772	3742
q14	452	423	410	410
q15	487	462	458	458
q16	734	683	682	682
q17	3822	3878	3931	3878
q18	8758	8784	8735	8735
q19	1729	1697	1694	1694
q20	2425	2096	2088	2088
q21	8583	8430	8483	8430
q22	1032	1010	982	982
Total cold run time: 84777 ms
Total hot run time: 79402 ms

@doris-robot
Copy link

doris-robot commented Jul 26, 2024

TeamCity be ut coverage result:
Function Coverage: 37.81% (8117/21466)
Line Coverage: 29.48% (66541/225743)
Region Coverage: 28.95% (34297/118468)
Branch Coverage: 24.83% (17623/70962)
Coverage Report: http://coverage.selectdb-in.cc/coverage/8b9392479b7c1e5e444dc6429bc93b563f37f449_8b9392479b7c1e5e444dc6429bc93b563f37f449/report/index.html

@doris-robot
Copy link

doris-robot commented Jul 26, 2024

TPC-DS: Total hot run time: 203735 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8b9392479b7c1e5e444dc6429bc93b563f37f449, data reload: false

query1	933	427	384	384
query2	6525	2852	2760	2760
query3	6956	204	199	199
query4	20087	17958	17887	17887
query5	19768	6477	6591	6477
query6	302	229	235	229
query7	4156	304	311	304
query8	454	398	399	398
query9	3154	2711	2620	2620
query10	424	307	299	299
query11	11269	10638	10865	10638
query12	130	76	74	74
query13	5607	715	703	703
query14	18020	13391	13691	13391
query15	363	245	241	241
query16	6453	298	262	262
query17	1690	1476	885	885
query18	2281	420	406	406
query19	217	150	150	150
query20	80	77	77	77
query21	196	91	93	91
query22	5258	4980	5041	4980
query23	32501	32076	32038	32038
query24	6978	6503	6599	6503
query25	525	421	427	421
query26	534	162	159	159
query27	1895	296	295	295
query28	6208	2399	2351	2351
query29	3036	2757	2880	2757
query30	248	167	170	167
query31	920	734	766	734
query32	69	65	60	60
query33	400	262	255	255
query34	850	480	469	469
query35	1127	955	893	893
query36	1523	1184	1140	1140
query37	85	64	59	59
query38	3058	2935	2950	2935
query39	1383	1333	1317	1317
query40	214	96	95	95
query41	47	45	44	44
query42	79	84	84	84
query43	670	676	729	676
query44	1165	722	728	722
query45	248	234	235	234
query46	1221	989	999	989
query47	1911	1676	1778	1676
query48	1023	723	706	706
query49	625	364	374	364
query50	868	601	588	588
query51	4762	4697	4679	4679
query52	90	78	81	78
query53	455	329	332	329
query54	2655	2465	2463	2463
query55	91	82	84	82
query56	247	208	197	197
query57	1185	1106	1090	1090
query58	226	202	213	202
query59	4372	4217	4132	4132
query60	236	216	205	205
query61	95	92	98	92
query62	846	469	484	469
query63	500	349	351	349
query64	2569	1495	1502	1495
query65	3618	3535	3593	3535
query66	811	379	382	379
query67	15802	15656	14800	14800
query68	10160	657	703	657
query69	580	340	345	340
query70	1745	1416	1557	1416
query71	416	315	319	315
query72	6577	3480	3470	3470
query73	739	325	331	325
query74	6348	5834	5814	5814
query75	5453	3695	3735	3695
query76	6430	1124	1196	1124
query77	1075	262	253	253
query78	12601	12465	11874	11874
query79	8755	627	628	627
query80	733	425	420	420
query81	488	234	241	234
query82	618	99	97	97
query83	173	132	131	131
query84	264	72	71	71
query85	730	333	336	333
query86	343	330	316	316
query87	3269	3038	2966	2966
query88	4180	2313	2330	2313
query89	405	286	321	286
query90	1933	214	209	209
query91	182	158	145	145
query92	56	50	54	50
query93	4206	576	567	567
query94	674	212	214	212
query95	1125	1069	1067	1067
query96	634	334	323	323
query97	6387	6324	6302	6302
query98	213	179	161	161
query99	2842	936	860	860
Total cold run time: 312628 ms
Total hot run time: 203735 ms

@doris-robot
Copy link

doris-robot commented Jul 26, 2024

ClickBench: Total hot run time: 30.72 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8b9392479b7c1e5e444dc6429bc93b563f37f449, data reload: false

query1	0.02	0.02	0.03
query2	0.07	0.02	0.02
query3	0.25	0.04	0.04
query4	1.78	0.06	0.06
query5	0.54	0.52	0.53
query6	1.30	0.59	0.61
query7	0.02	0.01	0.01
query8	0.03	0.03	0.02
query9	0.51	0.48	0.49
query10	0.53	0.54	0.53
query11	0.12	0.08	0.09
query12	0.12	0.09	0.09
query13	0.62	0.62	0.62
query14	0.76	0.80	0.78
query15	0.80	0.76	0.79
query16	0.39	0.38	0.36
query17	1.02	0.97	1.03
query18	0.22	0.26	0.25
query19	1.89	1.87	1.84
query20	0.02	0.01	0.01
query21	15.46	0.56	0.56
query22	2.05	1.87	2.00
query23	17.38	1.11	0.88
query24	7.58	0.78	1.06
query25	0.35	0.08	0.06
query26	0.77	0.15	0.16
query27	0.04	0.05	0.04
query28	5.75	0.76	0.77
query29	12.67	2.21	2.33
query30	0.61	0.55	0.52
query31	2.82	0.39	0.37
query32	3.36	0.50	0.52
query33	3.04	3.07	3.08
query34	15.27	4.80	4.80
query35	4.85	4.84	4.88
query36	1.06	1.02	1.01
query37	0.06	0.05	0.04
query38	0.04	0.02	0.02
query39	0.02	0.02	0.02
query40	0.16	0.14	0.15
query41	0.07	0.01	0.02
query42	0.03	0.01	0.02
query43	0.02	0.02	0.01
Total cold run time: 104.47 s
Total hot run time: 30.72 s

@doris-robot
Copy link

doris-robot commented Jul 26, 2024

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 8b9392479b7c1e5e444dc6429bc93b563f37f449 with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.2 seconds inserted 10000000 Rows, about 471K ops/s

lide-reed
lide-reed approved these changes Jul 29, 2024
View reviewed changes
Copy link
Contributor

@lide-reed lide-reed left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions
Copy link
Contributor

github-actions bot commented Jul 29, 2024

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Jul 29, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Jul 29, 2024

PR approved by anyone and no changes requested.

@cambyzju cambyzju merged commit 0ad9600 into apache:branch-2.0 Jul 30, 2024
mongo360 pushed a commit to mongo360/doris that referenced this pull request Dec 11, 2024
@cambyzju
[opt](mow) opt mow lookup with sequence column (apache#38287) (apache…
71eed82 
…#38407)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@lide-reed lide-reed lide-reed approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@cambyzju @doris-robot @lide-reed