Skip to content

Conversation

@xiaokang
Copy link
Contributor

@xiaokang xiaokang commented Feb 18, 2024

Proposed changes

related to #31035 #31051

  1. allow inverted index for key columns for MOR unique table
  2. allow inverted index without parser for value columns for MOR unique table, and do not push down predicate if it may cause wrong result
  3. disallow inverted index with parser for value columns for MOR unique table, due to MATCH without predicate push down is very slow.

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

@github-actions github-actions bot added area/planner Issues or PRs related to the query planner kind/test labels Feb 18, 2024
@xiaokang
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 50706 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 53b43c7f5bb9c51fbe4a1ce68c600eaea03f206c, data reload: false

------ Round 1 ----------------------------------
q1	17551	4355	4361	4355
q2	2028	157	140	140
q3	10271	1904	1948	1904
q4	10101	1267	1324	1267
q5	8477	4007	4030	4007
q6	238	124	125	124
q7	2011	1604	1594	1594
q8	9328	2771	2762	2762
q9	10822	10934	11165	10934
q10	8686	3549	3514	3514
q11	413	244	239	239
q12	469	299	299	299
q13	18332	3932	4019	3932
q14	360	316	332	316
q15	517	450	461	450
q16	699	584	600	584
q17	1151	979	1002	979
q18	7399	6890	6986	6890
q19	1703	1552	1572	1552
q20	545	315	321	315
q21	4471	4155	4157	4155
q22	495	394	400	394
Total cold run time: 116067 ms
Total hot run time: 50706 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4369	4297	4312	4297
q2	324	216	220	216
q3	4161	4176	4149	4149
q4	2763	2755	2776	2755
q5	7388	7192	7216	7192
q6	241	117	117	117
q7	3203	2806	2853	2806
q8	4364	4464	4474	4464
q9	17163	16845	16921	16845
q10	4275	4265	4293	4265
q11	774	688	678	678
q12	1025	852	859	852
q13	7384	3727	3755	3727
q14	440	432	425	425
q15	512	455	447	447
q16	756	700	703	700
q17	3901	3852	3868	3852
q18	8750	8824	8896	8824
q19	1743	1723	1660	1660
q20	2356	2183	2113	2113
q21	8523	8543	8527	8527
q22	1028	950	947	947
Total cold run time: 85443 ms
Total hot run time: 79858 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 240189 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 53b43c7f5bb9c51fbe4a1ce68c600eaea03f206c, data reload: false

query1	922	393	385	385
query2	6520	2213	2135	2135
query3	6919	201	202	201
query4	20268	18048	18088	18048
query5	19721	6500	6504	6500
query6	291	216	240	216
query7	4146	294	298	294
query8	269	224	223	223
query9	3185	2743	2650	2650
query10	415	275	319	275
query11	11416	10743	10711	10711
query12	126	72	72	72
query13	5576	657	617	617
query14	17757	13359	13610	13359
query15	359	222	228	222
query16	6437	273	268	268
query17	1701	1476	849	849
query18	2339	409	410	409
query19	205	150	149	149
query20	75	76	74	74
query21	184	104	96	96
query22	5432	5064	4818	4818
query23	32710	32038	31978	31978
query24	7021	6501	6454	6454
query25	515	439	420	420
query26	531	162	162	162
query27	1890	300	293	293
query28	6095	2174	2183	2174
query29	2908	2757	2691	2691
query30	236	160	167	160
query31	898	728	739	728
query32	69	62	60	60
query33	407	252	250	250
query34	854	469	490	469
query35	1110	900	922	900
query36	1641	1281	1343	1281
query37	86	61	60	60
query38	3063	2956	2915	2915
query39	1392	1321	1321	1321
query40	198	96	93	93
query41	34	31	30	30
query42	89	81	83	81
query43	610	568	595	568
query44	1118	710	727	710
query45	245	224	229	224
query46	1234	994	967	967
query47	1855	1734	1646	1646
query48	979	664	658	658
query49	609	367	379	367
query50	863	641	616	616
query51	5486	5433	5448	5433
query52	86	88	87	87
query53	441	330	321	321
query54	2635	2473	2505	2473
query55	98	73	83	73
query56	223	211	196	196
query57	1140	1123	1154	1123
query58	216	202	198	198
query59	3626	3069	3255	3069
query60	204	215	200	200
query61	91	84	86	84
query62	840	504	529	504
query63	468	342	336	336
query64	2380	1568	1478	1478
query65	3697	3561	3568	3561
query66	758	372	382	372
query67	17533	16898	15548	15548
query68	5997	655	653	653
query69	547	338	335	335
query70	1884	1757	1632	1632
query71	402	308	326	308
query72	6354	3436	3489	3436
query73	734	321	331	321
query74	6272	5920	5872	5872
query75	4529	3710	3647	3647
query76	3587	1187	1225	1187
query77	565	248	251	248
query78	32163	52717	48950	48950
query79	14193	644	620	620
query80	4451	382	388	382
query81	568	229	227	227
query82	771	90	94	90
query83	331	132	134	132
query84	261	67	67	67
query85	1820	292	288	288
query86	456	364	373	364
query87	3277	2992	3031	2992
query88	6570	2324	2351	2324
query89	392	283	270	270
query90	2505	210	215	210
query91	153	136	117	117
query92	64	50	48	48
query93	3726	591	554	554
query94	1682	205	208	205
query95	1110	1076	1065	1065
query96	656	337	326	326
query97	6473	6401	6323	6323
query98	198	175	176	175
query99	3757	967	918	918
Total cold run time: 339893 ms
Total hot run time: 240189 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.31 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 53b43c7f5bb9c51fbe4a1ce68c600eaea03f206c, data reload: false

query1	0.02	0.02	0.01
query2	0.06	0.02	0.02
query3	0.25	0.04	0.05
query4	1.82	0.08	0.06
query5	0.54	0.52	0.52
query6	1.34	0.62	0.61
query7	0.01	0.01	0.01
query8	0.04	0.02	0.02
query9	0.54	0.48	0.48
query10	0.52	0.53	0.52
query11	0.13	0.09	0.08
query12	0.12	0.09	0.10
query13	0.62	0.61	0.61
query14	0.78	0.81	0.78
query15	0.79	0.76	0.76
query16	0.37	0.37	0.36
query17	0.98	0.99	1.03
query18	0.22	0.27	0.23
query19	1.93	1.91	1.84
query20	0.02	0.01	0.01
query21	15.48	0.63	0.55
query22	2.18	2.22	1.18
query23	17.25	0.96	0.94
query24	6.33	1.25	1.02
query25	1.47	0.11	0.11
query26	0.33	0.16	0.14
query27	0.10	0.12	0.11
query28	7.97	0.73	0.71
query29	12.82	2.28	2.23
query30	0.59	0.55	0.53
query31	2.82	0.36	0.38
query32	3.44	0.49	0.50
query33	3.05	3.05	3.08
query34	15.26	4.80	4.77
query35	4.83	4.81	4.86
query36	1.06	1.02	1.03
query37	0.06	0.05	0.04
query38	0.04	0.02	0.02
query39	0.01	0.02	0.02
query40	0.16	0.14	0.15
query41	0.07	0.01	0.01
query42	0.02	0.01	0.01
query43	0.03	0.02	0.01
Total cold run time: 106.47 s
Total hot run time: 30.31 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 53b43c7f5bb9c51fbe4a1ce68c600eaea03f206c with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.9 seconds inserted 10000000 Rows, about 456K ops/s

@xiaokang xiaokang requested a review from qidaye February 18, 2024 10:10
Copy link
Member

@airborne12 airborne12 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions
Copy link
Contributor

PR approved by anyone and no changes requested.

Copy link
Contributor

@qidaye qidaye left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Feb 18, 2024
@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@xiaokang xiaokang merged commit a557fed into apache:branch-2.0 Feb 18, 2024
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. area/planner Issues or PRs related to the query planner kind/test reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants