Skip to content

[Fix](Variant) check column index validation for extracted columns #33766

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
eldenmoon merged 1 commit into from
Apr 18, 2024
Merged

[Fix](Variant) check column index validation for extracted columns #33766

eldenmoon merged 1 commit into from
Apr 18, 2024

Conversation

@eldenmoon
Copy link
Member

@eldenmoon eldenmoon commented Apr 17, 2024

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

doris-robot commented Apr 17, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@eldenmoon
Copy link
Member Author

eldenmoon commented Apr 17, 2024

run buildall

1 similar comment
@eldenmoon
Copy link
Member Author

eldenmoon commented Apr 17, 2024

run buildall

@github-actions
Copy link
Contributor

github-actions bot commented Apr 17, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

doris-robot commented Apr 17, 2024

TPC-H: Total hot run time: 38835 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e49a2a3372d8795b5fcff541a6de1459f62137ee, data reload: false

------ Round 1 ----------------------------------
q1	18425	4676	4365	4365
q2	2615	193	191	191
q3	10822	1248	1144	1144
q4	10556	810	742	742
q5	7537	2796	2708	2708
q6	224	139	140	139
q7	1043	616	614	614
q8	9234	2091	2052	2052
q9	7275	6672	6550	6550
q10	8630	3551	3539	3539
q11	441	227	234	227
q12	520	220	216	216
q13	17792	3003	2946	2946
q14	261	240	232	232
q15	544	481	482	481
q16	535	386	377	377
q17	977	641	671	641
q18	7501	6815	6714	6714
q19	4907	1527	1516	1516
q20	656	330	308	308
q21	3774	2831	3056	2831
q22	366	302	307	302
Total cold run time: 114635 ms
Total hot run time: 38835 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4375	4294	4461	4294
q2	379	286	273	273
q3	3028	2725	2735	2725
q4	1879	1545	1534	1534
q5	5377	5371	5315	5315
q6	208	126	122	122
q7	2254	1878	1902	1878
q8	3203	3342	3406	3342
q9	8593	8554	8585	8554
q10	3888	3715	3702	3702
q11	579	492	467	467
q12	766	594	574	574
q13	16798	2936	2966	2936
q14	299	274	278	274
q15	520	493	473	473
q16	487	423	449	423
q17	1794	1488	1475	1475
q18	7750	7412	7529	7412
q19	1659	1543	1546	1543
q20	1972	1760	1790	1760
q21	4960	4915	4854	4854
q22	549	441	469	441
Total cold run time: 71317 ms
Total hot run time: 54371 ms

@doris-robot
Copy link

doris-robot commented Apr 17, 2024

TeamCity be ut coverage result:
Function Coverage: 35.46% (8909/25123)
Line Coverage: 27.16% (73155/269335)
Region Coverage: 26.29% (37826/143862)
Branch Coverage: 23.08% (19270/83478)
Coverage Report: http://coverage.selectdb-in.cc/coverage/e49a2a3372d8795b5fcff541a6de1459f62137ee_e49a2a3372d8795b5fcff541a6de1459f62137ee/report/index.html

@doris-robot
Copy link

doris-robot commented Apr 17, 2024

TPC-DS: Total hot run time: 183923 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e49a2a3372d8795b5fcff541a6de1459f62137ee, data reload: false

query1	914	371	372	371
query2	6504	2568	2336	2336
query3	6670	209	206	206
query4	22855	21262	21365	21262
query5	4272	407	414	407
query6	262	175	174	174
query7	4587	287	282	282
query8	226	192	174	174
query9	8631	2306	2296	2296
query10	572	239	239	239
query11	14808	14165	14361	14165
query12	139	89	91	89
query13	1652	375	364	364
query14	9944	7865	7055	7055
query15	256	185	203	185
query16	8237	260	250	250
query17	1992	599	583	583
query18	2131	280	272	272
query19	325	155	193	155
query20	89	84	86	84
query21	195	124	122	122
query22	5063	4777	4816	4777
query23	33681	33014	33155	33014
query24	10827	2954	2853	2853
query25	635	367	372	367
query26	1341	145	143	143
query27	2936	303	303	303
query28	7488	1993	1978	1978
query29	895	609	587	587
query30	308	175	171	171
query31	959	715	735	715
query32	93	51	51	51
query33	749	243	243	243
query34	1052	470	511	470
query35	807	709	685	685
query36	1030	916	917	916
query37	131	70	70	70
query38	3311	3187	3217	3187
query39	1664	1568	1516	1516
query40	200	122	122	122
query41	46	43	44	43
query42	104	97	93	93
query43	568	553	550	550
query44	1180	713	711	711
query45	281	270	263	263
query46	1079	726	743	726
query47	1914	1867	1832	1832
query48	355	291	284	284
query49	997	372	363	363
query50	773	370	375	370
query51	6827	6802	6841	6802
query52	100	88	85	85
query53	340	274	269	269
query54	297	226	222	222
query55	74	68	71	68
query56	235	220	219	219
query57	1209	1130	1115	1115
query58	209	191	194	191
query59	3340	3169	3286	3169
query60	254	225	229	225
query61	92	87	89	87
query62	636	439	433	433
query63	300	274	273	273
query64	5484	3924	3999	3924
query65	3068	3057	3070	3057
query66	1425	338	324	324
query67	15389	14964	14927	14927
query68	5176	524	515	515
query69	494	290	297	290
query70	1215	1141	1134	1134
query71	1384	1262	1264	1262
query72	6440	2624	2464	2464
query73	697	318	318	318
query74	6805	6447	6455	6447
query75	3543	2629	2621	2621
query76	3461	1009	1037	1009
query77	541	256	260	256
query78	11016	10162	10246	10162
query79	3740	521	510	510
query80	1641	433	423	423
query81	535	244	246	244
query82	799	94	98	94
query83	316	171	167	167
query84	261	83	87	83
query85	1996	266	252	252
query86	491	295	289	289
query87	3524	3295	3315	3295
query88	4613	2389	2324	2324
query89	477	361	371	361
query90	2002	183	179	179
query91	124	97	96	96
query92	57	47	46	46
query93	5095	500	490	490
query94	1264	173	175	173
query95	392	297	292	292
query96	616	264	257	257
query97	3151	2936	2978	2936
query98	240	217	219	217
query99	1242	853	883	853
Total cold run time: 288355 ms
Total hot run time: 183923 ms

@doris-robot
Copy link

doris-robot commented Apr 17, 2024

ClickBench: Total hot run time: 30.36 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e49a2a3372d8795b5fcff541a6de1459f62137ee, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.03	0.04
query3	0.23	0.05	0.06
query4	1.67	0.08	0.08
query5	0.50	0.49	0.51
query6	1.48	0.72	0.71
query7	0.02	0.01	0.01
query8	0.05	0.04	0.04
query9	0.55	0.50	0.48
query10	0.54	0.57	0.54
query11	0.16	0.12	0.11
query12	0.14	0.12	0.12
query13	0.59	0.59	0.58
query14	0.75	0.78	0.79
query15	0.82	0.81	0.80
query16	0.36	0.38	0.37
query17	0.97	1.00	0.94
query18	0.20	0.25	0.24
query19	1.79	1.68	1.77
query20	0.01	0.01	0.02
query21	15.43	0.66	0.65
query22	3.89	7.46	1.97
query23	18.30	1.41	1.28
query24	1.88	0.26	0.21
query25	0.13	0.09	0.09
query26	0.26	0.15	0.16
query27	0.08	0.08	0.07
query28	13.26	0.99	0.97
query29	12.59	3.28	3.30
query30	0.26	0.06	0.07
query31	2.86	0.38	0.38
query32	3.26	0.46	0.45
query33	2.83	2.84	2.86
query34	17.19	4.39	4.41
query35	4.46	4.48	4.44
query36	0.65	0.46	0.46
query37	0.18	0.16	0.15
query38	0.15	0.14	0.14
query39	0.05	0.04	0.03
query40	0.17	0.14	0.13
query41	0.09	0.05	0.05
query42	0.05	0.05	0.04
query43	0.04	0.04	0.04
Total cold run time: 109.01 s
Total hot run time: 30.36 s

@doris-robot
Copy link

doris-robot commented Apr 17, 2024

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit e49a2a3372d8795b5fcff541a6de1459f62137ee with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      33 seconds loaded 861443392 Bytes, about 24 MB/s
Insert into select:       13.4 seconds inserted 10000000 Rows, about 746K ops/s

@eldenmoon eldenmoon added api-review Categorizes an issue or PR as actively needing an API review. variant and removed api-review Categorizes an issue or PR as actively needing an API review. labels Apr 17, 2024
xiaokang
xiaokang approved these changes Apr 18, 2024
View reviewed changes
Copy link
Contributor

@xiaokang xiaokang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Apr 18, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Apr 18, 2024

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Apr 18, 2024

PR approved by anyone and no changes requested.

@eldenmoon eldenmoon merged commit 14bbe31 into apache:master Apr 18, 2024
eldenmoon added a commit to eldenmoon/incubator-doris that referenced this pull request Apr 18, 2024
@eldenmoon
[Fix](Variant) check column index validation for extracted columns (a…
760eb60 
…pache#33766)
@eldenmoon eldenmoon mentioned this pull request Apr 18, 2024
Merged
yiguolei pushed a commit that referenced this pull request Apr 18, 2024
@eldenmoon
[Pick](Variant) pick #33734 #33766 #33707 to branch-2.1 (#33848)
8f6f4cf 
* [Fix](Variant Type) forbit distribution info contains variant columns (#33707)

* [Fix](Variant) VariantRootColumnIterator::read_by_rowids with wrong null map size (#33734)

insert_range_from should start from `size` with `count` elements for null map

* [Fix](Variant) check column index validation for extracted columns (#33766)
dataroaring pushed a commit to dataroaring/incubator-doris that referenced this pull request Apr 20, 2024
@eldenmoon @dataroaring
[Fix](Variant) check column index validation for extracted columns (a…
2d71f99 
…pache#33766)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@xiaokang xiaokang xiaokang approved these changes

@qidaye qidaye qidaye approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.0-merged dev/3.0.0-merged reviewed variant

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@eldenmoon @doris-robot @xiaokang @qidaye @yiguolei @dataroaring