Skip to content

[Fix](Variant) VariantRootColumnIterator::read_by_rowids with wrong null map size #33734

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
eldenmoon merged 1 commit into from
Apr 18, 2024
Merged

[Fix](Variant) VariantRootColumnIterator::read_by_rowids with wrong null map size #33734

eldenmoon merged 1 commit into from
Apr 18, 2024

Conversation

@eldenmoon
Copy link
Member

@eldenmoon eldenmoon commented Apr 16, 2024

…ull map size

insert_range_from should start from size with count elements for null map

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@eldenmoon
[Fix](Variant) VariantRootColumnIterator::read_by_rowids with wrong n…
152c54e 
…ull map size

insert_range_from should start from `size` with `count` elements for null map
@doris-robot
Copy link

doris-robot commented Apr 16, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@eldenmoon
Copy link
Member Author

eldenmoon commented Apr 16, 2024

run buildall

@eldenmoon eldenmoon added usercase Important user case type label variant labels Apr 16, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Apr 16, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

doris-robot commented Apr 16, 2024

TPC-H: Total hot run time: 38533 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 152c54e6c1a9b6e8aba177e6336d13ef05b0c3e5, data reload: false

------ Round 1 ----------------------------------
q1	17599	4358	4236	4236
q2	2016	185	179	179
q3	10594	1248	1192	1192
q4	10375	882	737	737
q5	7496	2732	2665	2665
q6	216	136	133	133
q7	1047	620	580	580
q8	9265	2064	2044	2044
q9	7392	6594	6534	6534
q10	8596	3525	3506	3506
q11	452	229	233	229
q12	427	222	213	213
q13	17760	2967	2940	2940
q14	268	226	229	226
q15	507	490	486	486
q16	512	381	380	380
q17	971	700	699	699
q18	7334	6826	6660	6660
q19	1609	1521	1511	1511
q20	641	320	306	306
q21	3458	2837	2776	2776
q22	369	301	308	301
Total cold run time: 108904 ms
Total hot run time: 38533 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4205	4143	4194	4143
q2	367	262	275	262
q3	3038	2713	2711	2711
q4	1886	1544	1601	1544
q5	5359	5394	5310	5310
q6	217	122	126	122
q7	2241	1911	1889	1889
q8	3224	3352	3276	3276
q9	8634	8582	8584	8582
q10	3910	3748	3678	3678
q11	580	494	472	472
q12	779	573	572	572
q13	17371	2946	2916	2916
q14	301	261	290	261
q15	508	472	479	472
q16	463	432	424	424
q17	1769	1496	1483	1483
q18	7647	7467	7587	7467
q19	1624	1549	1533	1533
q20	1964	1790	1753	1753
q21	4914	4895	4755	4755
q22	547	453	470	453
Total cold run time: 71548 ms
Total hot run time: 54078 ms

@doris-robot
Copy link

doris-robot commented Apr 16, 2024

TeamCity be ut coverage result:
Function Coverage: 35.49% (8906/25092)
Line Coverage: 27.21% (73126/268725)
Region Coverage: 26.34% (37809/143548)
Branch Coverage: 23.11% (19258/83332)
Coverage Report: http://coverage.selectdb-in.cc/coverage/152c54e6c1a9b6e8aba177e6336d13ef05b0c3e5_152c54e6c1a9b6e8aba177e6336d13ef05b0c3e5/report/index.html

@doris-robot
Copy link

doris-robot commented Apr 16, 2024

TPC-DS: Total hot run time: 184693 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 152c54e6c1a9b6e8aba177e6336d13ef05b0c3e5, data reload: false

query1	908	379	361	361
query2	6457	2596	2457	2457
query3	6664	204	205	204
query4	23755	21458	21298	21298
query5	4156	405	404	404
query6	278	176	168	168
query7	4583	294	283	283
query8	220	176	173	173
query9	8503	2343	2336	2336
query10	581	244	239	239
query11	14792	14190	14119	14119
query12	130	84	84	84
query13	1645	354	353	353
query14	10248	7723	7842	7723
query15	264	176	186	176
query16	8121	257	260	257
query17	1923	584	566	566
query18	2009	275	267	267
query19	325	149	151	149
query20	96	82	83	82
query21	194	127	122	122
query22	5013	4769	4779	4769
query23	33911	33285	33299	33285
query24	11744	3007	2881	2881
query25	649	357	369	357
query26	1746	153	150	150
query27	2887	307	313	307
query28	7296	1993	1982	1982
query29	1003	611	597	597
query30	271	173	172	172
query31	958	715	734	715
query32	97	54	55	54
query33	754	249	240	240
query34	1070	466	477	466
query35	810	671	685	671
query36	1065	887	886	886
query37	279	68	70	68
query38	3338	3227	3181	3181
query39	1593	1544	1540	1540
query40	281	123	123	123
query41	45	45	45	45
query42	98	94	93	93
query43	585	547	543	543
query44	1195	710	708	708
query45	273	258	253	253
query46	1067	706	723	706
query47	1900	1833	1834	1833
query48	352	298	289	289
query49	1208	373	361	361
query50	753	373	378	373
query51	6779	6632	6630	6630
query52	101	84	87	84
query53	343	276	267	267
query54	329	230	254	230
query55	80	74	69	69
query56	238	224	216	216
query57	1236	1112	1113	1112
query58	225	193	195	193
query59	3537	3296	3220	3220
query60	250	252	226	226
query61	89	87	91	87
query62	638	440	433	433
query63	308	275	273	273
query64	6443	3781	3763	3763
query65	3098	3019	3063	3019
query66	1368	345	351	345
query67	15261	15228	15124	15124
query68	5427	547	529	529
query69	525	297	304	297
query70	1259	1112	1187	1112
query71	1419	1265	1264	1264
query72	6447	2569	2440	2440
query73	724	324	318	318
query74	6748	6374	6479	6374
query75	3581	2571	2627	2571
query76	3881	951	921	921
query77	630	264	265	264
query78	10955	10181	10173	10173
query79	3743	520	509	509
query80	1955	427	422	422
query81	519	246	243	243
query82	1526	99	99	99
query83	327	171	168	168
query84	269	92	83	83
query85	1576	274	313	274
query86	470	311	301	301
query87	3459	3296	3301	3296
query88	4668	2384	2397	2384
query89	470	362	373	362
query90	1956	180	177	177
query91	122	96	97	96
query92	56	48	44	44
query93	4909	510	495	495
query94	1209	177	179	177
query95	383	303	284	284
query96	604	266	269	266
query97	3136	2952	2929	2929
query98	248	213	217	213
query99	1259	855	848	848
Total cold run time: 292876 ms
Total hot run time: 184693 ms

@doris-robot
Copy link

doris-robot commented Apr 16, 2024

ClickBench: Total hot run time: 30.25 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 152c54e6c1a9b6e8aba177e6336d13ef05b0c3e5, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.04	0.04
query3	0.23	0.06	0.07
query4	1.66	0.09	0.09
query5	0.50	0.50	0.52
query6	1.47	0.72	0.72
query7	0.02	0.01	0.02
query8	0.04	0.04	0.04
query9	0.56	0.49	0.49
query10	0.54	0.56	0.53
query11	0.16	0.12	0.12
query12	0.15	0.12	0.12
query13	0.60	0.60	0.59
query14	0.76	0.76	0.78
query15	0.84	0.81	0.81
query16	0.37	0.36	0.36
query17	0.94	0.95	0.98
query18	0.20	0.25	0.23
query19	1.78	1.70	1.67
query20	0.01	0.01	0.01
query21	15.41	0.65	0.65
query22	4.57	7.26	1.91
query23	18.29	1.36	1.21
query24	1.79	0.25	0.23
query25	0.14	0.09	0.07
query26	0.27	0.17	0.17
query27	0.08	0.08	0.08
query28	13.32	0.99	0.98
query29	12.58	3.28	3.24
query30	0.26	0.06	0.06
query31	2.85	0.40	0.39
query32	3.24	0.46	0.45
query33	2.82	2.78	2.78
query34	17.07	4.40	4.43
query35	4.48	4.45	4.43
query36	0.65	0.47	0.47
query37	0.17	0.15	0.15
query38	0.15	0.14	0.14
query39	0.04	0.05	0.04
query40	0.19	0.14	0.14
query41	0.10	0.05	0.05
query42	0.05	0.04	0.04
query43	0.04	0.04	0.04
Total cold run time: 109.51 s
Total hot run time: 30.25 s

@doris-robot
Copy link

doris-robot commented Apr 16, 2024

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 152c54e6c1a9b6e8aba177e6336d13ef05b0c3e5 with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       14.3 seconds inserted 10000000 Rows, about 699K ops/s

@xiaokang xiaokang changed the title [Fix](Variant) VariantRootColumnIterator::read_by_rowids with wrong n… [Fix](Variant) VariantRootColumnIterator::read_by_rowids with wrong null map size Apr 17, 2024
xiaokang
xiaokang approved these changes Apr 17, 2024
View reviewed changes
Copy link
Contributor

@xiaokang xiaokang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions
Copy link
Contributor

github-actions bot commented Apr 17, 2024

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Apr 17, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Apr 17, 2024

PR approved by anyone and no changes requested.

@eldenmoon eldenmoon merged commit 68caed8 into apache:master Apr 18, 2024
eldenmoon added a commit to eldenmoon/incubator-doris that referenced this pull request Apr 18, 2024
@eldenmoon
[Fix](Variant) VariantRootColumnIterator::read_by_rowids with wrong n…
9c2003f 
…ull map size (apache#33734)

insert_range_from should start from `size` with `count` elements for null map
yiguolei pushed a commit that referenced this pull request Apr 18, 2024
@eldenmoon @Doris-Extras
[Fix](Variant) VariantRootColumnIterator::read_by_rowids with wrong n…
04e30c9 
…ull map size (#33734)

insert_range_from should start from `size` with `count` elements for null map
@eldenmoon eldenmoon mentioned this pull request Apr 18, 2024
Merged
yiguolei pushed a commit that referenced this pull request Apr 18, 2024
@eldenmoon
[Pick](Variant) pick #33734 #33766 #33707 to branch-2.1 (#33848)
8f6f4cf 
* [Fix](Variant Type) forbit distribution info contains variant columns (#33707)

* [Fix](Variant) VariantRootColumnIterator::read_by_rowids with wrong null map size (#33734)

insert_range_from should start from `size` with `count` elements for null map

* [Fix](Variant) check column index validation for extracted columns (#33766)
dataroaring pushed a commit that referenced this pull request Apr 24, 2024
@eldenmoon @dataroaring
[Fix](Variant) VariantRootColumnIterator::read_by_rowids with wrong n…
9341cbb 
…ull map size (#33734)

insert_range_from should start from `size` with `count` elements for null map
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@xiaokang xiaokang xiaokang approved these changes

@qidaye qidaye qidaye approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. reviewed usercase Important user case type label variant

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@eldenmoon @doris-robot @xiaokang @qidaye