Skip to content

Conversation

@morningman
Copy link
Contributor

cherry pick from #42102

…hdown and implicit conversion pushdown (apache#42102)

Add a variable `enable_jdbc_cast_predicate_push_down`, the default value
is false, which prohibits the pushdown of non-constant predicates with
type conversion and all predicates with implicit conversion. This change
can prevent the wrong predicates from being pushed down to the Jdbc data
source, resulting in query data errors, because the predicates with cast
were not correctly pushed down to the data source before.
If you find that the data is read correctly and the performance is
better before this change, you can manually set this variable to true

```
| Expression                                          | Can Push Down |
|-----------------------------------------------------|---------------|
| column type equals const type                       | Yes           |
| column type equals cast const type                  | Yes           |
| cast column type equals const type                  | No            |
| cast column type equals cast const type             | No            |
| column type not equals column type                  | No            |
| column type not equals cast const type              | No            |
| cast column type not equals const type              | No            |
| cast column type not equals cast const type         | No            |

```
@morningman
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@doris-robot
Copy link

TPC-H: Total hot run time: 40864 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c7730b17de87b99876109481b06574eb3b22e082, data reload: false

------ Round 1 ----------------------------------
q1	17562	7722	7674	7674
q2	2053	163	143	143
q3	10751	1057	1123	1057
q4	10536	712	760	712
q5	7741	2823	2689	2689
q6	237	148	141	141
q7	950	599	598	598
q8	9357	1884	2004	1884
q9	6553	6407	6397	6397
q10	6942	2292	2263	2263
q11	446	257	257	257
q12	401	217	213	213
q13	17762	2979	2995	2979
q14	241	214	233	214
q15	572	519	514	514
q16	658	600	612	600
q17	957	550	588	550
q18	7160	6570	6642	6570
q19	2403	1056	1076	1056
q20	482	199	196	196
q21	4043	3301	3165	3165
q22	1093	999	992	992
Total cold run time: 108900 ms
Total hot run time: 40864 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7822	7543	7590	7543
q2	319	231	227	227
q3	3013	2930	2857	2857
q4	2063	1760	1766	1760
q5	5645	5671	5737	5671
q6	219	137	142	137
q7	2160	1757	1724	1724
q8	3364	3470	3354	3354
q9	8844	8805	8769	8769
q10	3569	3524	3568	3524
q11	596	491	491	491
q12	806	622	588	588
q13	16453	3145	3147	3145
q14	300	274	273	273
q15	563	515	532	515
q16	701	666	676	666
q17	1854	1615	1584	1584
q18	8120	7892	7398	7398
q19	3917	1584	1537	1537
q20	2095	1827	1846	1827
q21	5453	5284	5319	5284
q22	1149	1010	1043	1010
Total cold run time: 79025 ms
Total hot run time: 59884 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 193917 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c7730b17de87b99876109481b06574eb3b22e082, data reload: false

query1	1201	927	919	919
query2	6291	2066	2051	2051
query3	10916	3851	3807	3807
query4	68134	27696	23398	23398
query5	5768	458	444	444
query6	484	182	174	174
query7	6425	309	301	301
query8	339	237	227	227
query9	9525	2653	2621	2621
query10	534	272	260	260
query11	18326	15278	15720	15278
query12	159	105	105	105
query13	1619	407	416	407
query14	11359	6880	7274	6880
query15	228	176	175	175
query16	7234	527	470	470
query17	1063	579	554	554
query18	1825	302	295	295
query19	206	155	152	152
query20	111	107	113	107
query21	206	98	99	98
query22	4306	4130	4242	4130
query23	33935	33747	33841	33747
query24	5568	2874	2885	2874
query25	533	407	402	402
query26	691	158	166	158
query27	1699	290	294	290
query28	4394	2505	2501	2501
query29	675	438	430	430
query30	240	149	151	149
query31	998	794	816	794
query32	63	59	57	57
query33	438	273	288	273
query34	882	484	502	484
query35	825	725	771	725
query36	1050	940	917	917
query37	117	73	76	73
query38	3913	3853	3849	3849
query39	1490	1442	1448	1442
query40	201	99	99	99
query41	52	50	49	49
query42	109	100	99	99
query43	521	486	496	486
query44	1122	782	790	782
query45	184	165	161	161
query46	1119	700	702	700
query47	1889	1814	1807	1807
query48	476	383	375	375
query49	746	413	400	400
query50	804	406	411	406
query51	7215	7093	6997	6997
query52	105	94	88	88
query53	252	181	181	181
query54	563	446	445	445
query55	76	76	76	76
query56	240	235	230	230
query57	1194	1075	1084	1075
query58	208	200	216	200
query59	3126	2797	2969	2797
query60	271	257	248	248
query61	117	98	99	98
query62	775	671	692	671
query63	206	181	183	181
query64	1771	638	591	591
query65	3240	3156	3149	3149
query66	691	299	301	299
query67	15839	15467	15292	15292
query68	4605	547	542	542
query69	431	256	259	256
query70	1182	1129	1123	1123
query71	392	257	260	257
query72	6449	3873	3889	3873
query73	735	349	336	336
query74	10249	8836	8886	8836
query75	3367	2628	2603	2603
query76	2691	930	935	930
query77	493	271	258	258
query78	10610	9693	9576	9576
query79	8501	588	583	583
query80	1875	422	429	422
query81	560	235	243	235
query82	1286	116	118	116
query83	284	136	137	136
query84	285	79	81	79
query85	1825	292	271	271
query86	478	299	306	299
query87	4353	4270	4338	4270
query88	5589	2351	2362	2351
query89	556	288	293	288
query90	2035	182	175	175
query91	170	136	136	136
query92	65	46	51	46
query93	6643	546	538	538
query94	859	293	297	293
query95	347	246	256	246
query96	622	275	286	275
query97	3367	3108	3118	3108
query98	213	202	200	200
query99	1826	1260	1303	1260
Total cold run time: 339825 ms
Total hot run time: 193917 ms

@morningman morningman merged commit 92e01a5 into apache:branch-3.0 Nov 6, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants