Skip to content

branch-3.0: [Bug](runtime-filter) fix missing rf when target not exist on backend where the producer is… #47108 #47247

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
BiteTheDDDDt merged 1 commit into from
Jan 21, 2025
Merged

branch-3.0: [Bug](runtime-filter) fix missing rf when target not exist on backend where the producer is… #47108 #47247

BiteTheDDDDt merged 1 commit into from
Jan 21, 2025

Conversation

@github-actions
Copy link
Contributor

@github-actions github-actions bot commented Jan 20, 2025

Cherry-picked from #47108

@BiteTheDDDDt
[Bug](runtime-filter) fix missing rf when target not exist on backend…
8dbbfd9 
… where the producer is… (#47108)

… located

### What problem does this PR solve?
Sometimes the consumer of a certain rf does not exist on the BE where
the producer of a certain rf is located. At this time, the send to
remote operation of the rf will cause problems and the rf will be lost.

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [x] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [x] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [x] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
@github-actions github-actions bot requested a review from dataroaring as a code owner January 20, 2025 07:53
@Thearas
Copy link
Contributor

Thearas commented Jan 20, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring dataroaring reopened this Jan 20, 2025
@Thearas
Copy link
Contributor

Thearas commented Jan 20, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Jan 20, 2025

TPC-H: Total hot run time: 40912 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8dbbfd938ad3fac2b7bc13a1f8fbd46644099a70, data reload: false

------ Round 1 ----------------------------------
q1	17565	7667	7334	7334
q2	2049	188	164	164
q3	10559	1092	1180	1092
q4	10463	733	698	698
q5	7751	2878	2875	2875
q6	240	148	149	148
q7	978	609	609	609
q8	9357	1986	1973	1973
q9	6556	6412	6430	6412
q10	7047	2293	2308	2293
q11	467	280	259	259
q12	398	212	209	209
q13	17793	2997	3004	2997
q14	245	210	216	210
q15	562	522	532	522
q16	679	617	622	617
q17	971	616	523	523
q18	7278	6720	6773	6720
q19	1395	1094	1002	1002
q20	476	209	196	196
q21	4065	3106	3229	3106
q22	1074	987	953	953
Total cold run time: 107968 ms
Total hot run time: 40912 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7366	7284	7170	7170
q2	327	236	236	236
q3	2933	2953	2964	2953
q4	2063	1832	1837	1832
q5	5695	5844	5772	5772
q6	221	143	147	143
q7	2241	1833	1853	1833
q8	3349	3541	3498	3498
q9	8816	8883	8826	8826
q10	3585	3537	3532	3532
q11	638	514	504	504
q12	803	623	619	619
q13	7712	3214	3205	3205
q14	299	292	301	292
q15	584	520	525	520
q16	728	699	674	674
q17	1837	1632	1606	1606
q18	8419	7831	7589	7589
q19	1666	1509	1552	1509
q20	2086	1904	1859	1859
q21	5539	5416	5368	5368
q22	1177	1004	1011	1004
Total cold run time: 68084 ms
Total hot run time: 60544 ms

@doris-robot
Copy link

doris-robot commented Jan 20, 2025

TPC-DS: Total hot run time: 197985 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8dbbfd938ad3fac2b7bc13a1f8fbd46644099a70, data reload: false

query1	1293	935	937	935
query2	6216	2093	2063	2063
query3	10854	4266	4217	4217
query4	66124	29071	23630	23630
query5	4926	456	447	447
query6	409	171	164	164
query7	5578	319	317	317
query8	313	227	229	227
query9	8901	2696	2695	2695
query10	447	269	266	266
query11	17381	15203	15719	15203
query12	159	109	102	102
query13	1513	428	433	428
query14	10654	6884	7795	6884
query15	208	179	190	179
query16	7184	486	518	486
query17	1123	593	624	593
query18	1802	337	331	331
query19	213	166	167	166
query20	127	111	112	111
query21	217	105	117	105
query22	4613	4717	4397	4397
query23	34789	34195	34415	34195
query24	6281	2937	2990	2937
query25	543	420	459	420
query26	660	174	183	174
query27	1864	346	372	346
query28	4534	2507	2448	2448
query29	710	465	460	460
query30	251	179	164	164
query31	984	850	840	840
query32	72	54	55	54
query33	437	288	288	288
query34	901	504	555	504
query35	827	720	752	720
query36	1102	978	984	978
query37	121	78	74	74
query38	4050	3988	4005	3988
query39	1539	1469	1496	1469
query40	205	102	103	102
query41	52	53	49	49
query42	114	103	100	100
query43	543	499	500	499
query44	1220	841	835	835
query45	186	173	170	170
query46	1164	751	728	728
query47	2016	1908	1914	1908
query48	472	388	389	388
query49	717	400	391	391
query50	858	440	434	434
query51	7204	7306	7133	7133
query52	102	88	89	88
query53	270	182	187	182
query54	570	460	454	454
query55	77	80	75	75
query56	261	271	240	240
query57	1263	1143	1119	1119
query58	211	206	208	206
query59	3255	2862	3025	2862
query60	276	250	253	250
query61	106	129	122	122
query62	777	657	650	650
query63	221	196	205	196
query64	1393	660	628	628
query65	3285	3207	3194	3194
query66	707	311	306	306
query67	15875	15785	15793	15785
query68	4128	594	570	570
query69	440	266	281	266
query70	1183	1082	1131	1082
query71	371	264	259	259
query72	6424	4113	4054	4054
query73	762	362	360	360
query74	10247	8990	8999	8990
query75	3372	2661	2700	2661
query76	1927	1174	1111	1111
query77	489	276	273	273
query78	10585	9724	9616	9616
query79	2049	611	635	611
query80	1396	427	424	424
query81	534	243	243	243
query82	1265	119	118	118
query83	164	150	148	148
query84	286	84	76	76
query85	998	307	297	297
query86	432	313	289	289
query87	4423	4369	4378	4369
query88	3748	2425	2345	2345
query89	457	288	298	288
query90	1922	190	188	188
query91	181	151	145	145
query92	66	54	52	52
query93	2962	550	548	548
query94	870	290	305	290
query95	361	253	263	253
query96	636	280	281	280
query97	3396	3201	3189	3189
query98	211	206	201	201
query99	1580	1298	1290	1290
Total cold run time: 321185 ms
Total hot run time: 197985 ms

@doris-robot
Copy link

doris-robot commented Jan 20, 2025

ClickBench: Total hot run time: 33.66 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8dbbfd938ad3fac2b7bc13a1f8fbd46644099a70, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.03	0.03
query3	0.24	0.06	0.06
query4	1.63	0.10	0.10
query5	0.53	0.51	0.53
query6	1.13	0.73	0.74
query7	0.02	0.02	0.02
query8	0.05	0.03	0.03
query9	0.57	0.52	0.51
query10	0.55	0.54	0.55
query11	0.15	0.10	0.10
query12	0.14	0.13	0.11
query13	0.60	0.60	0.59
query14	3.12	2.92	2.95
query15	0.90	0.83	0.83
query16	0.38	0.40	0.37
query17	1.04	1.02	1.04
query18	0.24	0.23	0.22
query19	1.85	1.89	2.09
query20	0.01	0.01	0.01
query21	15.70	0.62	0.60
query22	2.59	2.29	1.94
query23	17.03	0.95	0.98
query24	2.89	1.22	1.80
query25	0.20	0.15	0.12
query26	0.51	0.14	0.14
query27	0.04	0.04	0.04
query28	9.81	1.11	1.08
query29	12.57	3.30	3.28
query30	0.25	0.07	0.06
query31	2.85	0.40	0.38
query32	3.24	0.47	0.47
query33	3.05	3.02	3.02
query34	17.06	4.47	4.46
query35	4.51	4.50	4.50
query36	0.67	0.48	0.49
query37	0.10	0.06	0.07
query38	0.04	0.04	0.03
query39	0.04	0.03	0.02
query40	0.16	0.12	0.13
query41	0.08	0.03	0.02
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 106.71 s
Total hot run time: 33.66 s

@BiteTheDDDDt BiteTheDDDDt merged commit b0d5f91 into branch-3.0 Jan 21, 2025
21 of 22 checks passed
@github-actions github-actions bot deleted the auto-pick-47108-branch-3.0 branch January 21, 2025 07:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dataroaring dataroaring Awaiting requested review from dataroaring

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@Thearas @doris-robot @BiteTheDDDDt @dataroaring