[GLUTEN-8538][VL] Fix incorrect calculation of buffer size by jkhaliqi · Pull Request #8542 · apache/gluten

jkhaliqi · 2025-01-15T17:04:57Z

What changes were proposed in this pull request?

Fix incorrect calculation of buffer size
False positives:
cpp/velox/udf/UdfLoader.cc:79
cpp/velox/udf/UdfLoader.cc:106

(Fixes: #8538)

github-actions · 2025-01-15T17:05:14Z

FelixYBW · 2025-01-15T20:49:13Z

 bool StdMemoryAllocator::allocate(int64_t size, void** out) {
+  if (size < 0) {
+    return false;
+  }


Let's use assert

FelixYBW · 2025-01-15T20:57:15Z

+  size_t totalSize = nmemb * size;
+  if (totalSize / size != nmemb || nmemb < 0 || size < 0) {
+    return false;
+  }


isn't nmemb<0 || size <0 enough?

use assert instead

Right I was thinking we had to check to make sure there was no overflow but if calloc fails then it will give nullptr anyways and return false so the sizing check being bigger than 0 should be enough!

FelixYBW · 2025-01-16T00:35:03Z

 }

 bool StdMemoryAllocator::allocate(int64_t size, void** out) {
+  assert(size < 0);


GLUTEN_CHECK

FelixYBW · 2025-01-16T00:35:11Z

 }

 bool StdMemoryAllocator::allocateZeroFilled(int64_t nmemb, int64_t size, void** out) {
+  assert(nmemb < 0 && size < 0);


GLUTEN_CHECK

FelixYBW · 2025-01-16T00:35:17Z

 }

 bool StdMemoryAllocator::allocateAligned(uint64_t alignment, int64_t size, void** out) {
+  assert(size < 0);


GLUTEN_CHECK

GlutenPerfBot · 2025-01-18T17:55:04Z

===== Performance report for TPCDS SF2000 with Velox backend, for reference only ====

query	log/native_master_01_18_2025_time.csv	log/native_master_01_17_2025_764ca97547_time.csv	difference	percentage
q1	15.89	15.74	-0.150	99.05%
q2	15.38	15.74	0.363	102.36%
q3	5.74	4.51	-1.236	78.48%
q4	86.00	86.19	0.191	100.22%
q5	11.73	13.36	1.622	113.83%
q6	5.36	5.16	-0.198	96.31%
q7	5.95	8.37	2.418	140.61%
q8	5.91	4.34	-1.562	73.56%
q9	27.10	28.04	0.941	103.47%
q10	13.43	11.97	-1.457	89.15%
q11	43.23	41.60	-1.628	96.23%
q12	2.39	2.10	-0.293	87.75%
q13	9.19	11.25	2.056	122.37%
q14a	61.25	66.90	5.650	109.22%
q14b	56.10	56.74	0.639	101.14%
q15	3.43	3.54	0.113	103.30%
q16	29.18	29.20	0.015	100.05%
q17	7.49	7.58	0.086	101.15%
q18	9.72	10.21	0.489	105.03%
q19	4.02	3.72	-0.294	92.68%
q20	2.48	2.22	-0.260	89.50%
q21	1.65	1.82	0.170	110.25%
q22	9.42	9.71	0.286	103.04%
q23a	135.51	137.85	2.345	101.73%
q23b	162.63	163.36	0.728	100.45%
q24a	105.16	107.60	2.437	102.32%
q24b	99.11	93.90	-5.204	94.75%
q25	7.03	6.41	-0.616	91.24%
q26	5.08	4.22	-0.854	83.18%
q27	5.23	6.58	1.347	125.73%
q28	39.37	36.67	-2.701	93.14%
q29	17.17	16.98	-0.197	98.86%
q30	6.44	6.35	-0.085	98.68%
q31	10.32	11.02	0.704	106.82%
q32	2.20	2.19	-0.004	99.81%
q33	6.98	7.01	0.027	100.39%
q34	4.50	4.93	0.436	109.70%
q35	10.44	9.82	-0.623	94.03%
q36	5.52	5.68	0.163	102.95%
q37	5.32	5.31	-0.010	99.81%
q38	17.16	17.40	0.234	101.36%
q39a	5.75	4.38	-1.375	76.09%
q39b	4.94	4.77	-0.172	96.52%
q40	5.27	5.61	0.341	106.47%
q41	0.86	1.14	0.276	132.06%
q42	1.37	1.36	-0.010	99.28%
q43	4.67	4.56	-0.111	97.62%
q44	12.56	12.74	0.182	101.45%
q45	4.38	7.33	2.958	167.58%
q46	5.67	5.40	-0.274	95.17%
q47	19.69	20.61	0.924	104.69%
q48	6.62	6.68	0.064	100.97%
q49	11.08	10.89	-0.183	98.35%
q50	38.29	38.76	0.470	101.23%
q51	14.05	14.60	0.552	103.93%
q52	1.23	1.25	0.026	102.08%
q53	3.00	2.93	-0.079	97.38%
q54	6.64	6.85	0.215	103.24%
q55	1.42	1.33	-0.095	93.35%
q56	7.41	7.22	-0.184	97.51%
q57	14.15	13.04	-1.108	92.17%
q58	3.33	3.39	0.058	101.73%
q59	6.47	6.76	0.289	104.47%
q60	7.76	7.62	-0.141	98.18%
q61	7.56	8.17	0.607	108.03%
q62	5.33	5.34	0.011	100.21%
q63	3.12	3.16	0.041	101.32%
q64	64.07	62.41	-1.652	97.42%
q65	30.52	29.81	-0.705	97.69%
q66	4.18	4.42	0.243	105.81%
q67	225.83	227.89	2.061	100.91%
q68	4.31	4.41	0.098	102.28%
q69	7.09	7.10	0.016	100.22%
q70	12.71	12.78	0.071	100.56%
q71	5.57	5.75	0.177	103.18%
q72	37.98	37.66	-0.321	99.15%
q73	3.28	3.38	0.105	103.19%
q74	26.62	26.63	0.014	100.05%
q75	44.33	43.63	-0.703	98.41%
q76	14.06	14.42	0.359	102.55%
q77	3.27	3.33	0.060	101.82%
q78	84.77	84.94	0.167	100.20%
q79	5.06	5.26	0.204	104.04%
q80	16.45	17.01	0.560	103.41%
q81	8.08	7.90	-0.181	97.76%
q82	10.34	10.18	-0.156	98.49%
q83	2.87	2.51	-0.367	87.23%
q84	3.58	3.70	0.118	103.29%
q85	9.87	10.02	0.154	101.56%
q86	4.65	4.55	-0.104	97.77%
q87	19.60	20.44	0.849	104.33%
q88	23.08	23.13	0.046	100.20%
q89	4.90	4.38	-0.520	89.37%
q90	3.96	3.55	-0.407	89.71%
q91	6.13	6.51	0.384	106.27%
q92	2.30	1.88	-0.420	81.74%
q93	51.89	52.03	0.141	100.27%
q94	17.59	18.00	0.413	102.35%
q9	97.96	98.15	0.186	100.19%
q5	3.34	3.01	-0.322	90.35%
q96	28.77	28.01	-0.758	97.36%
q97	3.06	3.10	0.042	101.39%
q98	10.01	10.38	0.371	103.70%
q99	10.01	10.38	0.371	103.70%
total	2181.94	2191.53	9.595	100.44%

GlutenPerfBot · 2025-01-18T18:49:37Z

===== Performance report for TPCH SF2000 with Velox backend, for reference only ====

query	log/native_master_01_18_2025_time.csv	log/native_master_01_16_2025_ac8e03a1d6_time.csv	difference	percentage
q1	42.21	44.15	1.944	104.61%
q2	44.01	44.79	0.780	101.77%
q3	93.24	91.65	-1.584	98.30%
q4	70.58	71.60	1.016	101.44%
q5	182.05	183.36	1.314	100.72%
q6	19.55	18.70	-0.856	95.62%
q7	104.58	106.97	2.391	102.29%
q8	186.45	183.81	-2.644	98.58%
q9	279.64	282.15	2.511	100.90%
q10	104.35	101.26	-3.094	97.04%
q11	33.52	33.99	0.465	101.39%
q12	41.99	42.06	0.071	100.17%
q13	76.84	77.28	0.444	100.58%
q14	34.16	37.06	2.903	108.50%
q15	65.54	67.07	1.521	102.32%
q16	26.73	26.74	0.008	100.03%
q17	233.96	232.20	-1.758	99.25%
q18	353.22	345.10	-8.120	97.70%
q19	36.86	38.97	2.104	105.71%
q20	59.87	59.76	-0.116	99.81%
q21	530.40	530.71	0.312	100.06%
q22	24.18	24.05	-0.128	99.47%
total	2643.92	2643.41	-0.514	99.98%

GlutenPerfBot · 2025-01-19T17:54:26Z

===== Performance report for TPCDS SF2000 with Velox backend, for reference only ====

query	log/native_master_01_19_2025_time.csv	log/native_master_01_18_2025_bc25ac1ea3_time.csv	difference	percentage
q1	16.63	15.89	-0.741	95.54%
q2	15.26	15.38	0.116	100.76%
q3	3.07	5.74	2.670	186.91%
q4	85.98	86.00	0.020	100.02%
q5	13.67	11.73	-1.933	85.86%
q6	4.83	5.36	0.530	110.96%
q7	7.69	5.95	-1.738	77.40%
q8	4.40	5.91	1.505	134.18%
q9	28.72	27.10	-1.620	94.36%
q10	11.62	13.43	1.815	115.62%
q11	42.24	43.23	0.997	102.36%
q12	2.18	2.39	0.213	109.80%
q13	10.48	9.19	-1.292	87.68%
q14a	69.94	61.25	-8.697	87.57%
q14b	55.84	56.10	0.256	100.46%
q15	3.72	3.43	-0.285	92.34%
q16	29.69	29.18	-0.510	98.28%
q17	7.48	7.49	0.011	100.15%
q18	10.17	9.72	-0.448	95.59%
q19	3.91	4.02	0.105	102.69%
q20	2.19	2.48	0.285	112.97%
q21	1.97	1.65	-0.314	84.04%
q22	10.24	9.42	-0.821	91.99%
q23a	145.37	135.51	-9.862	93.22%
q23b	163.17	162.63	-0.541	99.67%
q24a	109.91	105.16	-4.752	95.68%
q24b	102.62	99.11	-3.509	96.58%
q25	6.08	7.03	0.942	115.48%
q26	4.17	5.08	0.911	121.86%
q27	4.94	5.23	0.292	105.92%
q28	36.65	39.37	2.726	107.44%
q29	18.58	17.17	-1.404	92.44%
q30	6.55	6.44	-0.113	98.27%
q31	10.39	10.32	-0.070	99.33%
q32	2.31	2.20	-0.118	94.89%
q33	7.76	6.98	-0.777	89.98%
q34	4.53	4.50	-0.033	99.27%
q35	11.32	10.44	-0.881	92.22%
q36	5.69	5.52	-0.176	96.90%
q37	5.42	5.32	-0.094	98.26%
q38	24.17	17.16	-7.001	71.03%
q39a	4.91	5.75	0.839	117.07%
q39b	4.44	4.94	0.503	111.34%
q40	5.88	5.27	-0.616	89.53%
q41	0.86	0.86	-0.003	99.70%
q42	1.31	1.37	0.068	105.21%
q43	4.48	4.67	0.188	104.19%
q44	12.97	12.56	-0.415	96.80%
q45	4.59	4.38	-0.217	95.28%
q46	5.32	5.67	0.349	106.55%
q47	20.10	19.69	-0.414	97.94%
q48	6.46	6.62	0.155	102.40%
q49	10.74	11.08	0.343	103.19%
q50	38.61	38.29	-0.323	99.16%
q51	14.21	14.05	-0.163	98.86%
q52	1.42	1.23	-0.192	86.46%
q53	3.29	3.00	-0.285	91.33%
q54	7.85	6.64	-1.207	84.61%
q55	1.34	1.42	0.080	105.98%
q56	7.45	7.41	-0.045	99.40%
q57	12.14	14.15	2.013	116.58%
q58	3.46	3.33	-0.127	96.32%
q59	6.55	6.47	-0.075	98.85%
q60	8.08	7.76	-0.325	95.98%
q61	7.62	7.56	-0.057	99.25%
q62	5.16	5.33	0.170	103.30%
q63	3.24	3.12	-0.119	96.32%
q64	59.83	64.07	4.235	107.08%
q65	30.28	30.52	0.236	100.78%
q66	4.71	4.18	-0.530	88.75%
q67	227.33	225.83	-1.497	99.34%
q68	4.51	4.31	-0.198	95.61%
q69	8.31	7.09	-1.218	85.34%
q70	13.15	12.71	-0.442	96.64%
q71	6.03	5.57	-0.454	92.48%
q72	38.48	37.98	-0.504	98.69%
q73	3.47	3.28	-0.188	94.56%
q74	26.75	26.62	-0.130	99.51%
q75	43.18	44.33	1.151	102.67%
q76	14.36	14.06	-0.301	97.90%
q77	3.51	3.27	-0.237	93.24%
q78	84.37	84.77	0.394	100.47%
q79	5.23	5.06	-0.175	96.65%
q80	16.98	16.45	-0.533	96.86%
q81	8.17	8.08	-0.092	98.87%
q82	9.75	10.34	0.587	106.02%
q83	2.43	2.87	0.446	118.41%
q84	3.67	3.58	-0.092	97.50%
q85	10.37	9.87	-0.502	95.16%
q86	4.43	4.65	0.224	105.06%
q87	17.23	19.60	2.365	113.73%
q88	24.43	23.08	-1.353	94.46%
q89	4.51	4.90	0.391	108.68%
q90	3.77	3.96	0.191	105.06%
q91	5.26	6.13	0.872	116.59%
q92	4.15	2.30	-1.852	55.41%
q93	53.21	51.89	-1.318	97.52%
q94	15.78	17.59	1.816	111.51%
q9	94.50	97.96	3.467	103.67%
q5	3.11	3.34	0.222	107.14%
q96	28.29	28.77	0.484	101.71%
q97	2.84	3.06	0.220	107.75%
q98	10.49	10.01	-0.478	95.44%
q99	10.49	10.01	-0.478	95.44%
total	2210.94	2181.94	-29.004	98.69%

GlutenPerfBot · 2025-01-19T18:49:15Z

===== Performance report for TPCH SF2000 with Velox backend, for reference only ====

query	log/native_master_01_19_2025_time.csv	log/native_master_01_18_2025_bc25ac1ea3_time.csv	difference	percentage
q1	43.83	42.21	-1.626	96.29%
q2	45.19	44.01	-1.181	97.39%
q3	91.23	93.24	2.011	102.20%
q4	70.46	70.58	0.119	100.17%
q5	181.02	182.05	1.027	100.57%
q6	19.58	19.55	-0.025	99.87%
q7	103.92	104.58	0.661	100.64%
q8	186.53	186.45	-0.081	99.96%
q9	281.41	279.64	-1.773	99.37%
q10	102.39	104.35	1.964	101.92%
q11	33.61	33.52	-0.084	99.75%
q12	42.64	41.99	-0.654	98.47%
q13	76.76	76.84	0.071	100.09%
q14	35.90	34.16	-1.742	95.15%
q15	67.46	65.54	-1.913	97.16%
q16	26.49	26.73	0.236	100.89%
q17	232.91	233.96	1.050	100.45%
q18	360.20	353.22	-6.980	98.06%
q19	37.50	36.86	-0.641	98.29%
q20	60.61	59.87	-0.739	98.78%
q21	535.28	530.40	-4.881	99.09%
q22	25.12	24.18	-0.935	96.28%
total	2660.04	2643.92	-16.117	99.39%

security volation check fix

github-actions bot added the VELOX label Jan 15, 2025

jkhaliqi force-pushed the jk_cve_buffer_size branch 2 times, most recently from 360af1c to d5d3f55 Compare January 15, 2025 17:13

FelixYBW reviewed Jan 15, 2025

View reviewed changes

jkhaliqi force-pushed the jk_cve_buffer_size branch 3 times, most recently from 6ad6908 to b72a002 Compare January 15, 2025 22:04

FelixYBW reviewed Jan 16, 2025

View reviewed changes

jkhaliqi force-pushed the jk_cve_buffer_size branch 3 times, most recently from bcc1d66 to 498ae2d Compare January 16, 2025 16:33

[GLUTEN-8538][VL] Fix incorrect calculation of buffer size

e51bcee

jkhaliqi force-pushed the jk_cve_buffer_size branch from 498ae2d to e51bcee Compare January 17, 2025 18:48

FelixYBW approved these changes Jan 17, 2025

View reviewed changes

FelixYBW merged commit bc25ac1 into apache:main Jan 17, 2025

baibaichen pushed a commit to baibaichen/gluten that referenced this pull request Feb 1, 2025

[GLUTEN-8538][VL] Fix incorrect calculation of buffer size (apache#8542)

e6766ef

security volation check fix

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[GLUTEN-8538][VL] Fix incorrect calculation of buffer size#8542

[GLUTEN-8538][VL] Fix incorrect calculation of buffer size#8542
FelixYBW merged 1 commit intoapache:mainfrom
jkhaliqi:jk_cve_buffer_size

jkhaliqi commented Jan 15, 2025

Uh oh!

github-actions bot commented Jan 15, 2025

Uh oh!

FelixYBW Jan 15, 2025

Uh oh!

FelixYBW Jan 15, 2025

Uh oh!

jkhaliqi Jan 15, 2025

Uh oh!

FelixYBW Jan 16, 2025

Uh oh!

FelixYBW Jan 16, 2025

Uh oh!

FelixYBW Jan 16, 2025

Uh oh!

GlutenPerfBot commented Jan 18, 2025

Uh oh!

GlutenPerfBot commented Jan 18, 2025

Uh oh!

GlutenPerfBot commented Jan 19, 2025

Uh oh!

GlutenPerfBot commented Jan 19, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

jkhaliqi commented Jan 15, 2025

What changes were proposed in this pull request?

Uh oh!

github-actions bot commented Jan 15, 2025

Uh oh!

FelixYBW Jan 15, 2025

Choose a reason for hiding this comment

Uh oh!

FelixYBW Jan 15, 2025

Choose a reason for hiding this comment

Uh oh!

jkhaliqi Jan 15, 2025

Choose a reason for hiding this comment

Uh oh!

FelixYBW Jan 16, 2025

Choose a reason for hiding this comment

Uh oh!

FelixYBW Jan 16, 2025

Choose a reason for hiding this comment

Uh oh!

FelixYBW Jan 16, 2025

Choose a reason for hiding this comment

Uh oh!

GlutenPerfBot commented Jan 18, 2025

Uh oh!

GlutenPerfBot commented Jan 18, 2025

Uh oh!

GlutenPerfBot commented Jan 19, 2025

Uh oh!

GlutenPerfBot commented Jan 19, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants