Skip to content

Conversation

@924060929
Copy link
Contributor

@924060929 924060929 commented May 8, 2025

What problem does this PR solve?

fix backend_active_tasks table only return one backend's data, for example:

select sum(SCAN_ROWS), sum(SCAN_BYTES)
from backend_active_tasks 
where QUERY_ID = 'd299cb2156ef4870-aea578938f703503'

the root cause is: the partition column be_id has be pruned, and can not get scan ranges from multiple backends, this bug is introduced by #11266

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented May 8, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@924060929
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 33813 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8a8a8fe3c9b77f29f46d6ee716c40f0e4d0bca97, data reload: false

------ Round 1 ----------------------------------
q1	25916	5101	4996	4996
q2	2054	279	189	189
q3	10406	1260	691	691
q4	10266	987	522	522
q5	8207	2409	2372	2372
q6	187	173	135	135
q7	924	740	615	615
q8	9327	1311	1127	1127
q9	6817	5046	5159	5046
q10	6857	2311	1895	1895
q11	486	292	270	270
q12	340	345	212	212
q13	17789	3696	3044	3044
q14	229	227	218	218
q15	540	481	490	481
q16	430	434	364	364
q17	583	857	358	358
q18	7573	7227	7160	7160
q19	1492	964	555	555
q20	335	326	216	216
q21	4076	3367	2368	2368
q22	1111	1014	979	979
Total cold run time: 115945 ms
Total hot run time: 33813 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5138	5082	5068	5068
q2	238	332	224	224
q3	2186	2650	2305	2305
q4	1315	1797	1368	1368
q5	4543	4467	4337	4337
q6	213	168	124	124
q7	1984	1897	1766	1766
q8	2584	2610	2500	2500
q9	7228	7235	7192	7192
q10	2981	3165	2724	2724
q11	582	495	486	486
q12	694	779	594	594
q13	3513	3859	3278	3278
q14	274	293	286	286
q15	526	490	476	476
q16	429	468	449	449
q17	1157	1536	1387	1387
q18	7688	7617	7294	7294
q19	815	825	1024	825
q20	1973	2067	1875	1875
q21	5098	4712	4601	4601
q22	1048	1018	981	981
Total cold run time: 52207 ms
Total hot run time: 50140 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 185381 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8a8a8fe3c9b77f29f46d6ee716c40f0e4d0bca97, data reload: false

query1	1035	467	493	467
query2	6560	1802	1757	1757
query3	6754	230	219	219
query4	26238	23706	23389	23389
query5	4341	625	472	472
query6	302	200	180	180
query7	4614	486	276	276
query8	308	262	219	219
query9	8582	2538	2525	2525
query10	468	316	273	273
query11	15618	15011	14845	14845
query12	160	106	104	104
query13	1649	522	419	419
query14	8595	6073	6200	6073
query15	204	186	170	170
query16	7145	613	471	471
query17	1012	708	567	567
query18	1957	407	292	292
query19	185	178	151	151
query20	119	110	117	110
query21	211	123	102	102
query22	4218	4164	4054	4054
query23	34037	32929	33005	32929
query24	8406	2387	2382	2382
query25	531	440	382	382
query26	1225	272	160	160
query27	2738	492	333	333
query28	4353	2076	2054	2054
query29	754	546	449	449
query30	284	213	196	196
query31	925	861	790	790
query32	75	68	65	65
query33	568	372	342	342
query34	798	839	534	534
query35	784	792	731	731
query36	941	995	901	901
query37	112	101	79	79
query38	4205	4196	4109	4109
query39	1437	1397	1415	1397
query40	218	139	120	120
query41	63	63	72	63
query42	121	112	111	111
query43	507	501	454	454
query44	1257	808	801	801
query45	191	177	190	177
query46	813	1032	658	658
query47	1756	1778	1703	1703
query48	370	410	314	314
query49	793	510	461	461
query50	643	689	410	410
query51	4138	4220	4079	4079
query52	107	104	101	101
query53	226	249	172	172
query54	574	566	500	500
query55	82	120	82	82
query56	299	311	296	296
query57	1128	1140	1085	1085
query58	263	248	257	248
query59	2605	2658	2447	2447
query60	320	326	305	305
query61	130	127	128	127
query62	772	707	670	670
query63	215	191	185	185
query64	4348	1004	679	679
query65	4382	4256	4281	4256
query66	1142	408	323	323
query67	15635	15936	15202	15202
query68	8385	869	512	512
query69	465	299	265	265
query70	1204	1106	1105	1105
query71	506	317	290	290
query72	5819	4743	4782	4743
query73	714	609	339	339
query74	8774	8847	8654	8654
query75	3818	3207	2735	2735
query76	3632	1173	741	741
query77	788	374	288	288
query78	10041	9915	9303	9303
query79	2925	808	567	567
query80	631	510	451	451
query81	457	254	217	217
query82	451	126	101	101
query83	290	248	285	248
query84	291	115	83	83
query85	799	349	321	321
query86	333	276	280	276
query87	4384	4416	4275	4275
query88	2853	2185	2193	2185
query89	422	330	283	283
query90	1921	211	212	211
query91	141	139	181	139
query92	79	62	56	56
query93	1825	946	580	580
query94	675	405	300	300
query95	375	290	290	290
query96	480	557	271	271
query97	3307	3234	3100	3100
query98	222	209	205	205
query99	1423	1422	1250	1250
Total cold run time: 273976 ms
Total hot run time: 185381 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 29.35 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8a8a8fe3c9b77f29f46d6ee716c40f0e4d0bca97, data reload: false

query1	0.04	0.04	0.03
query2	0.13	0.10	0.10
query3	0.26	0.19	0.19
query4	1.62	0.19	0.19
query5	0.59	0.55	0.56
query6	1.19	0.71	0.71
query7	0.02	0.02	0.01
query8	0.05	0.04	0.03
query9	0.57	0.51	0.50
query10	0.58	0.57	0.57
query11	0.15	0.11	0.11
query12	0.15	0.12	0.12
query13	0.62	0.60	0.60
query14	0.78	0.80	0.81
query15	0.87	0.84	0.86
query16	0.38	0.37	0.39
query17	1.04	1.01	0.99
query18	0.21	0.20	0.20
query19	1.86	1.79	1.81
query20	0.01	0.01	0.02
query21	15.40	0.95	0.56
query22	0.76	1.28	0.91
query23	14.72	1.40	0.64
query24	7.22	1.10	0.75
query25	0.51	0.26	0.06
query26	0.55	0.16	0.13
query27	0.05	0.05	0.05
query28	9.56	0.86	0.44
query29	12.55	3.88	3.24
query30	0.25	0.09	0.06
query31	2.81	0.59	0.38
query32	3.22	0.56	0.48
query33	3.02	3.02	3.01
query34	15.74	5.11	4.57
query35	4.59	4.57	4.56
query36	0.66	0.49	0.48
query37	0.08	0.06	0.06
query38	0.05	0.04	0.03
query39	0.03	0.02	0.02
query40	0.17	0.15	0.14
query41	0.08	0.03	0.02
query42	0.04	0.02	0.02
query43	0.03	0.04	0.03
Total cold run time: 103.21 s
Total hot run time: 29.35 s

@morningman morningman added dev/2.1.x dev/3.0.x usercase Important user case type label labels May 9, 2025
@github-actions
Copy link
Contributor

github-actions bot commented May 9, 2025

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels May 9, 2025
@github-actions
Copy link
Contributor

github-actions bot commented May 9, 2025

PR approved by anyone and no changes requested.

@924060929 924060929 merged commit 6f817a7 into apache:master May 9, 2025
31 of 33 checks passed
@924060929 924060929 deleted the master-fix-metadata-table-scan-one-be branch May 9, 2025 03:30
github-actions bot pushed a commit that referenced this pull request May 9, 2025
…one backend's data (#50721)

fix backend_active_tasks table only return one backend's data, for
example:
```sql
select sum(SCAN_ROWS), sum(SCAN_BYTES)
from backend_active_tasks 
where QUERY_ID = 'd299cb2156ef4870-aea578938f703503'
```

the root cause is: the partition column `be_id` has be pruned, and can
not get scan ranges from multiple backends, this bug is introduced by #11266
github-actions bot pushed a commit that referenced this pull request May 9, 2025
…one backend's data (#50721)

fix backend_active_tasks table only return one backend's data, for
example:
```sql
select sum(SCAN_ROWS), sum(SCAN_BYTES)
from backend_active_tasks 
where QUERY_ID = 'd299cb2156ef4870-aea578938f703503'
```

the root cause is: the partition column `be_id` has be pruned, and can
not get scan ranges from multiple backends, this bug is introduced by #11266
yiguolei pushed a commit that referenced this pull request May 9, 2025
@yiguolei yiguolei mentioned this pull request May 13, 2025
dataroaring pushed a commit that referenced this pull request May 13, 2025
koarz pushed a commit to koarz/doris that referenced this pull request Jun 4, 2025
…one backend's data (apache#50721)

fix backend_active_tasks table only return one backend's data, for
example:
```sql
select sum(SCAN_ROWS), sum(SCAN_BYTES)
from backend_active_tasks 
where QUERY_ID = 'd299cb2156ef4870-aea578938f703503'
```

the root cause is: the partition column `be_id` has be pruned, and can
not get scan ranges from multiple backends, this bug is introduced by apache#11266
@gavinchou gavinchou mentioned this pull request Jun 11, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.10-merged dev/3.0.6-merged reviewed usercase Important user case type label

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants