Skip to content

branch-3.0: [fix](core) fix subreplace when inputting a large number of empty strings #49241 #49302

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dataroaring merged 1 commit into from
Mar 25, 2025
Merged

branch-3.0: [fix](core) fix subreplace when inputting a large number of empty strings #49241 #49302

dataroaring merged 1 commit into from
Mar 25, 2025

Conversation

@github-actions
Copy link
Contributor

@github-actions github-actions bot commented Mar 20, 2025

Cherry-picked from #49241

@Mryange
[fix](core) fix subreplace when inputting a large number of empty str…
790848c 
…ings (#49241)

### What problem does this PR solve?

```
    #0 0x55f5cd170097 in doris::validate_ascii_fast(char const*, unsigned long) /mnt/disk12/yanxuecheng/tmp-doris/be/src/util/simd/vstring_function.h:60:37
    #1 0x55f5cd16e3c0 in doris::simd::VStringFunctions::is_ascii(doris::StringRef const&) /mnt/disk12/yanxuecheng/tmp-doris/be/src/util/simd/vstring_function.h:212:16
    #2 0x55f5cd16dcd3 in auto doris::vectorized::SubReplaceImpl::replace_execute(doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long)::'lambda'(auto, auto, auto, auto)::operator()<std::integral_constant<bool, false>, std::integral_constant<bool, false>, std::integral_constant<bool, false>, std::integral_constant<bool, false>>(auto, auto, auto, auto) const /mnt/disk12/yanxuecheng/tmp-doris/be/src/vec/functions/function_string.h:3600:25
    #3 0x55f5cd16d926 in auto std::__invoke_impl<void, doris::vectorized::SubReplaceImpl::replace_execute(doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long)::'lambda'(auto, auto, auto, auto), std::integral_constant<bool, false>, std::integral_constant<bool, false>, std::integral_constant<bool, false>, std::integral_constant<bool, false>>(std::__invoke_other, auto&&, std::integral_constant<bool, false>&&, std::integral_constant<bool, false>&&, std::integral_constant<bool, false>&&, std::integral_constant<bool, false>&&) /mnt/disk6/common/ldb_toolchain_robin/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:61:14
    #4 0x55f5cd16d8b6 in std::__invoke_result<auto, std::integral_constant<bool, false>, std::integral_constant<bool, false>, std::integral_constant<bool, false>, std::integral_constant<bool, false>>::type std::__invoke<doris::vectorized::SubReplaceImpl::replace_execute(doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long)::'lambda'(auto, auto, auto, auto), std::integral_constant<bool, false>, std::integral_constant<bool, false>, std::integral_constant<bool, false>, std::integral_constant<bool, false>>(auto&&, std::integral_constant<bool, false>&&, std::integral_constant<bool, false>&&, std::integral_constant<bool, false>&&, std::integral_constant<bool, false>&&) /mnt/disk6/common/ldb_toolchain_robin/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:96:14
    #5 0x55f5cd16cfed in std::__detail::__variant::__gen_vtable_impl<std::__detail::__variant::_Multi_array<std::__detail::__variant::__deduce_visit_result<void> (*)(doris::vectorized::SubReplaceImpl::replace_execute(doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long)::'lambda'(auto, auto, auto, auto)&&, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>&&, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>&&, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>&&, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>&&)>, std::integer_sequence<unsigned long, 0ul, 0ul, 0ul, 0ul>>::__visit_invoke(doris::vectorized::SubReplaceImpl::replace_execute(doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long)::'lambda'(auto, auto, auto, auto)&&, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>&&, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>&&, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>&&, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>&&) /mnt/disk6/common/ldb_toolchain_robin/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/variant:1032:11
    #6 0x55f5cd16ce46 in decltype(auto) std::__do_visit<std::__detail::__variant::__deduce_visit_result<void>, doris::vectorized::SubReplaceImpl::replace_execute(doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long)::'lambda'(auto, auto, auto, auto), std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>>(auto&&, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>&&, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>&&, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>&&, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>&&) /mnt/disk6/common/ldb_toolchain_robin/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/variant:1758:15
    #7 0x55f5cd16c7ca in std::invoke_result<auto, std::__conditional<is_lvalue_reference_v<std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>>>::type<std::variant_alternative<0ul, std::remove_reference<decltype(__variant::__as(std::declval<std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>>()))>::type>::type&, std::variant_alternative<0ul, std::remove_reference<decltype(__variant::__as(std::declval<std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>>()))>::type>::type&&>, std::__conditional<is_lvalue_reference_v<std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>>>::type<std::variant_alternative<0ul, std::remove_reference<decltype(__variant::__as(std::declval<std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>>()))>::type>::type&, std::variant_alternative<0ul, std::remove_reference<decltype(__variant::__as(std::declval<std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>>()))>::type>::type&&>, std::__conditional<is_lvalue_reference_v<std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>>>::type<std::variant_alternative<0ul, std::remove_reference<decltype(__variant::__as(std::declval<std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>>()))>::type>::type&, std::variant_alternative<0ul, std::remove_reference<decltype(__variant::__as(std::declval<std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>>()))>::type>::type&&>, std::__conditional<is_lvalue_reference_v<std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>>>::type<std::variant_alternative<0ul, std::remove_reference<decltype(__variant::__as(std::declval<std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>>()))>::type>::type&, std::variant_alternative<0ul, std::remove_reference<decltype(__variant::__as(std::declval<std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>>()))>::type>::type&&>>::type std::visit<doris::vectorized::SubReplaceImpl::replace_execute(doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long)::'lambda'(auto, auto, auto, auto), std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>>(auto&&, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>&&, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>&&, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>&&, std::variant<std::integral_constant<bool, false>, std::integral_constant<bool, true>>&&) /mnt/disk6/common/ldb_toolchain_robin/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/variant:1859:9
    #8 0x55f5cd16bd75 in doris::vectorized::SubReplaceImpl::replace_execute(doris::vectorized::Block&, std::vector<unsigned int, std::allocator<unsigned int>> const&, unsigned int, unsigned long) /mnt/disk12/yanxuecheng/tmp-doris/be/src/vec/functions/function_string.h:3594:9
```

Related PR:(#40929)
@github-actions github-actions bot requested a review from dataroaring as a code owner March 20, 2025 06:19
@Thearas
Copy link
Contributor

Thearas commented Mar 20, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring dataroaring reopened this Mar 20, 2025
@Thearas
Copy link
Contributor

Thearas commented Mar 20, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Mar 20, 2025

TPC-H: Total hot run time: 40260 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 790848c60bd050ed2374b86ea322bbaa60f0b230, data reload: false

------ Round 1 ----------------------------------
q1	17587	6777	6605	6605
q2	2077	167	166	166
q3	10588	1083	1177	1083
q4	10212	820	786	786
q5	7766	2931	2906	2906
q6	219	138	131	131
q7	994	623	608	608
q8	9355	1967	2065	1967
q9	6685	6468	6434	6434
q10	7004	2250	2311	2250
q11	470	270	262	262
q12	405	221	216	216
q13	17781	2973	3061	2973
q14	240	214	206	206
q15	509	465	462	462
q16	656	598	596	596
q17	993	638	561	561
q18	7325	6643	6593	6593
q19	1453	1103	1103	1103
q20	476	208	206	206
q21	4018	3191	3244	3191
q22	1072	999	955	955
Total cold run time: 107885 ms
Total hot run time: 40260 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6623	6596	6582	6582
q2	333	243	224	224
q3	2913	2843	2979	2843
q4	2039	1892	1793	1793
q5	5724	5773	5746	5746
q6	205	128	131	128
q7	2234	1899	1824	1824
q8	3450	3590	3577	3577
q9	8845	9007	8906	8906
q10	3579	3573	3549	3549
q11	612	496	502	496
q12	788	615	607	607
q13	9675	3240	3146	3146
q14	310	294	264	264
q15	516	468	469	468
q16	690	648	643	643
q17	1874	1665	1589	1589
q18	8238	7728	7613	7613
q19	1707	1604	1516	1516
q20	2036	1898	1895	1895
q21	5593	5378	5472	5378
q22	1123	1040	995	995
Total cold run time: 69107 ms
Total hot run time: 59782 ms

@doris-robot
Copy link

doris-robot commented Mar 20, 2025

BE UT Coverage Report

Increment line coverage 100.00% (2/2) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 38.89% (10171/26155)
Line Coverage 30.33% (86791/286152)
Region Coverage 29.36% (44570/151790)
Branch Coverage 25.89% (22672/87582)

@doris-robot
Copy link

doris-robot commented Mar 20, 2025

TPC-DS: Total hot run time: 197850 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 790848c60bd050ed2374b86ea322bbaa60f0b230, data reload: false

query1	1311	922	933	922
query2	6239	2075	2026	2026
query3	10829	4576	4422	4422
query4	61209	28424	23281	23281
query5	5214	460	453	453
query6	398	184	169	169
query7	5513	316	310	310
query8	305	235	227	227
query9	8808	2690	2666	2666
query10	455	297	258	258
query11	18266	15150	15859	15150
query12	182	110	105	105
query13	1450	473	451	451
query14	9835	7639	7442	7442
query15	200	176	194	176
query16	7225	498	538	498
query17	1170	593	609	593
query18	1895	323	333	323
query19	226	167	156	156
query20	119	114	112	112
query21	202	107	106	106
query22	4704	4588	4472	4472
query23	34654	34263	33948	33948
query24	6163	2997	2967	2967
query25	544	439	416	416
query26	653	171	175	171
query27	1919	356	388	356
query28	4190	2525	2450	2450
query29	734	490	456	456
query30	252	173	165	165
query31	994	841	852	841
query32	66	55	56	55
query33	440	309	339	309
query34	952	519	531	519
query35	896	763	733	733
query36	1104	940	973	940
query37	121	69	65	65
query38	4063	4079	3970	3970
query39	1547	1475	1463	1463
query40	204	101	96	96
query41	49	47	47	47
query42	117	101	106	101
query43	547	506	495	495
query44	1214	828	871	828
query45	190	167	176	167
query46	1172	750	749	749
query47	2010	1906	1895	1895
query48	477	382	389	382
query49	743	414	390	390
query50	863	450	444	444
query51	7481	7347	7252	7252
query52	103	95	97	95
query53	270	187	185	185
query54	591	472	481	472
query55	84	81	79	79
query56	289	262	263	262
query57	1269	1201	1168	1168
query58	213	209	217	209
query59	3185	2953	2778	2778
query60	287	271	265	265
query61	110	107	107	107
query62	792	655	645	645
query63	221	192	185	185
query64	1505	664	656	656
query65	3268	3243	3218	3218
query66	699	298	303	298
query67	16000	15405	15495	15405
query68	4224	590	587	587
query69	429	276	275	275
query70	1159	1152	1051	1051
query71	348	291	259	259
query72	6356	4036	4083	4036
query73	777	359	361	359
query74	10545	8952	8974	8952
query75	3364	2683	2691	2683
query76	2094	1114	1092	1092
query77	485	270	281	270
query78	10645	9622	9667	9622
query79	1242	611	616	611
query80	824	449	445	445
query81	494	241	233	233
query82	1287	91	92	91
query83	167	144	145	144
query84	287	90	73	73
query85	856	303	293	293
query86	319	305	307	305
query87	4446	4329	4326	4326
query88	3567	2494	2425	2425
query89	416	303	290	290
query90	2018	189	194	189
query91	183	150	147	147
query92	62	50	49	49
query93	1284	563	566	563
query94	745	292	304	292
query95	362	258	263	258
query96	622	290	295	290
query97	3323	3194	3153	3153
query98	218	202	201	201
query99	1784	1271	1289	1271
Total cold run time: 313932 ms
Total hot run time: 197850 ms

@doris-robot
Copy link

doris-robot commented Mar 20, 2025

ClickBench: Total hot run time: 32.87 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 790848c60bd050ed2374b86ea322bbaa60f0b230, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.04	0.03
query3	0.24	0.07	0.06
query4	1.62	0.11	0.11
query5	0.52	0.52	0.52
query6	1.14	0.74	0.73
query7	0.02	0.02	0.02
query8	0.03	0.05	0.04
query9	0.56	0.51	0.51
query10	0.57	0.56	0.56
query11	0.15	0.11	0.10
query12	0.14	0.11	0.11
query13	0.61	0.59	0.60
query14	2.73	2.76	2.85
query15	0.89	0.83	0.82
query16	0.38	0.39	0.38
query17	0.97	1.02	1.01
query18	0.24	0.22	0.22
query19	1.99	1.80	2.09
query20	0.02	0.01	0.01
query21	15.36	0.60	0.58
query22	2.57	2.34	2.81
query23	16.99	0.98	0.96
query24	3.02	1.35	0.97
query25	0.24	0.14	0.22
query26	0.50	0.13	0.13
query27	0.03	0.04	0.04
query28	10.11	0.50	0.48
query29	12.57	3.20	3.23
query30	0.24	0.06	0.05
query31	2.85	0.39	0.38
query32	3.29	0.45	0.46
query33	2.97	2.99	3.06
query34	17.13	4.44	4.49
query35	4.55	4.52	4.53
query36	0.69	0.49	0.48
query37	0.09	0.07	0.06
query38	0.05	0.04	0.04
query39	0.03	0.02	0.02
query40	0.17	0.14	0.13
query41	0.08	0.02	0.03
query42	0.04	0.02	0.02
query43	0.04	0.03	0.02
Total cold run time: 106.54 s
Total hot run time: 32.87 s

dataroaring
dataroaring approved these changes Mar 25, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit fcdb79c into branch-3.0 Mar 25, 2025
22 of 24 checks passed
@github-actions github-actions bot deleted the auto-pick-49241-branch-3.0 branch March 25, 2025 06:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dataroaring dataroaring dataroaring approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@Thearas @doris-robot @dataroaring @Mryange