Set DataFusion runtime configurations through SQL interface by kumarlokesh · Pull Request #15594 · apache/datafusion

kumarlokesh · 2025-04-05T15:28:52Z

Which issue does this PR close?

Closes Set DataFusion runtime configurations through SQL interface #15552.

Rationale for this change

What changes are included in this PR?

Extended the session context to handle runtime configurations.
Currently support memory limit configuration by using SET datafusion.runtime.memory_limit = '100M'.
Updated documentation.

Are these changes tested?

Are there any user-facing changes?

berkaysynnada · 2025-04-06T12:28:18Z

Hello @kumarlokesh. Thank you for working on this. I have 2 questions/concerns. Let's discuss on them a bit to get a future-proof design

There are also runtime_env: Arc<RuntimeEnv> in SessionState and TaskContext structs, and this RuntimeEnv has a field of pub memory_pool: Arc<dyn MemoryPool>. These MemoryPool implementations have some configurations about memory limit as well. Keeping 2 different things for the same purposes always bring trouble. So, I wonder can we somehow connect those 2 points.
Do you plan to practice this configuration in any part of the execution? While bringing such a feature, implementing at least one use-case makes things more clear and becomes a guidance for the further developments.

kumarlokesh · 2025-04-07T20:08:08Z

Hello @kumarlokesh. Thank you for working on this. I have 2 questions/concerns. Let's discuss on them a bit to get a future-proof design

There are also runtime_env: Arc<RuntimeEnv> in SessionState and TaskContext structs, and this RuntimeEnv has a field of pub memory_pool: Arc<dyn MemoryPool>. These MemoryPool implementations have some configurations about memory limit as well. Keeping 2 different things for the same purposes always bring trouble. So, I wonder can we somehow connect those 2 points.

Do you plan to practice this configuration in any part of the execution? While bringing such a feature, implementing at least one use-case makes things more clear and becomes a guidance for the further developments.

@berkaysynnada Thank you for the feedback!

Have revised the PR to ensure runtime environment configuration like memory-limit is exposed through the already defined RuntimeEnv -> memory_pool setting.
As a user can set the configuration datafusion.runtime.memory_limit (and other available runtime environment configurations) through SQL interface, I think having tests around this behaviour should be a good starting point. Can't think of another good use case besides this. Thoughts?

2010YOUY01 · 2025-04-08T05:03:04Z

datafusion/core/tests/sql/runtime_config.rs

+    let ctx = SessionContext::new();
+
+    // Set memory limit to 100MB using SQL - note the quotes around the value
+    ctx.sql("SET datafusion.runtime.memory_limit = '100M'")


Running this test can't ensure this config is set, because it can run without setting the memory limit and use the default UnboundedMemoryPool, instead I think it should assert the spill count for a query that has spilled some intermediate data.

SET datafusion.runtime.memory_limit = '1M' set datafusion.execution.sort_spill_reservation_bytes = 0; select * from generate_series(1, 100000) as t1(v1) order by v1; -- And assert spill-count from the query is > 0

You can check this

datafusion/datafusion/core/tests/memory_limit/mod.rs

Line 525 in 4ae0ede

let spill_count = metrics.spill_count().unwrap();

for how to assert the spill file count.

Later after all configurations are added, I think we should make this test case stronger by setting more runtime configs, and do some property test to ensure all of them are properly set.

BTW, I tried the above test locally, and the generate_series UDTF seems not registered in the PR branch, but it works in the main branch.

Running this test can't ensure this config is set, because it can run without setting the memory limit and use the default UnboundedMemoryPool, instead I think it should assert the spill count for a query that has spilled some intermediate data.

SET datafusion.runtime.memory_limit = '1M' set datafusion.execution.sort_spill_reservation_bytes = 0; select * from generate_series(1, 100000) as t1(v1) order by v1; -- And assert spill-count from the query is > 0

You can check this

datafusion/datafusion/core/tests/memory_limit/mod.rs

Line 525 in 4ae0ede

let spill_count = metrics.spill_count().unwrap();

for how to assert the spill file count.
Later after all configurations are added, I think we should make this test case stronger by setting more runtime configs, and do some property test to ensure all of them are properly set.

BTW, I tried the above test locally, and the generate_series UDTF seems not registered in the PR branch, but it works in the main branch.

@2010YOUY01 @berkaysynnada addressed above in 6501f21.

berkaysynnada · 2025-04-08T14:12:30Z

datafusion/core/src/execution/context/mod.rs

+
+    /// Parse memory limit from string to number of bytes
+    /// e.g. '1.5G', '100M'
+    fn parse_memory_limit(&self, limit: &str) -> Result<usize> {


Maybe we can open this function to external uses?

berkaysynnada · 2025-04-08T14:17:16Z

Thank you @kumarlokesh for addressing my comments. I don't have further suggestions or concern other than @2010YOUY01 shared

2010YOUY01

Thank you. I plan to leave this PR open for 2 days more before merging, in case anyone has additional comments.

2010YOUY01 · 2025-04-14T05:22:45Z

datafusion/core/src/execution/context/mod.rs

+
+                let mut state = self.state.write();
+                let mut builder =
+                    RuntimeEnvBuilder::from_runtime_env(state.runtime_env());


This implementation is all cloning Arcs inside RuntimeEnv, I think it shouldn't have any issue.

alamb · 2025-04-15T18:38:42Z

Thanks -- I recommend we merge this after the 47.0.0 release is created (likely tomorrow)

alamb

This looks really nice -- thank you @kumarlokesh and @2010YOUY01

alamb · 2025-04-16T19:32:05Z

We made branch-47 so let's get this one in!

…-configs-through-sql

alamb · 2025-04-16T19:32:26Z

I merged up from main to rerun CI to be sure all tests pass

alamb · 2025-04-17T20:13:01Z

🚀 so exciting -- thank you

…5594) * Set DataFusion runtime configurations through SQL interface * fix clippy warnings * use spill count based tests for checking applied memory limit --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

github-actions bot added documentation Improvements or additions to documentation core Core DataFusion crate labels Apr 5, 2025

kumarlokesh force-pushed the allow-setting-runtime-configs-through-sql branch 3 times, most recently from 3d743e5 to bfb1d0e Compare April 5, 2025 17:14

github-actions bot added the common Related to common crate label Apr 5, 2025

kumarlokesh force-pushed the allow-setting-runtime-configs-through-sql branch from bfb1d0e to 8d894be Compare April 5, 2025 17:43

github-actions bot added the sqllogictest SQL Logic Tests (.slt) label Apr 5, 2025

kumarlokesh mentioned this pull request Apr 5, 2025

Set DataFusion runtime configurations through SQL interface #15552

Closed

alamb added the enhancement New feature or request label Apr 7, 2025

kumarlokesh force-pushed the allow-setting-runtime-configs-through-sql branch from 8d894be to fa34e62 Compare April 7, 2025 18:43

github-actions bot added development-process Related to development process of DataFusion execution Related to the execution crate and removed sqllogictest SQL Logic Tests (.slt) common Related to common crate labels Apr 7, 2025

kumarlokesh force-pushed the allow-setting-runtime-configs-through-sql branch 5 times, most recently from bae6d2f to 08d038c Compare April 7, 2025 19:40

2010YOUY01 reviewed Apr 8, 2025

View reviewed changes

berkaysynnada reviewed Apr 8, 2025

View reviewed changes

kumarlokesh added 3 commits April 12, 2025 19:09

Set DataFusion runtime configurations through SQL interface

7de97f0

fix clippy warnings

fdf6b5b

use spill count based tests for checking applied memory limit

6501f21

kumarlokesh force-pushed the allow-setting-runtime-configs-through-sql branch from 08d038c to 6501f21 Compare April 12, 2025 17:09

kumarlokesh requested a review from 2010YOUY01 April 12, 2025 17:43

kumarlokesh requested a review from berkaysynnada April 12, 2025 17:43

2010YOUY01 approved these changes Apr 14, 2025

View reviewed changes

alamb approved these changes Apr 15, 2025

View reviewed changes

Merge remote-tracking branch 'apache/main' into allow-setting-runtime…

3ca3be2

…-configs-through-sql

2010YOUY01 merged commit 2cba3ad into apache:main Apr 17, 2025
29 checks passed

kumarlokesh deleted the allow-setting-runtime-configs-through-sql branch April 17, 2025 05:38

2010YOUY01 mentioned this pull request Jul 26, 2025

Make the temporary directory (for spills) configurable through configuration API #16921

Closed

Conversation

kumarlokesh commented Apr 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

berkaysynnada commented Apr 6, 2025

Uh oh!

kumarlokesh commented Apr 7, 2025

Uh oh!

2010YOUY01 Apr 8, 2025

Choose a reason for hiding this comment

Uh oh!

kumarlokesh Apr 12, 2025

Choose a reason for hiding this comment

Uh oh!

berkaysynnada Apr 8, 2025

Choose a reason for hiding this comment

Uh oh!

berkaysynnada commented Apr 8, 2025

Uh oh!

2010YOUY01 left a comment

Choose a reason for hiding this comment

Uh oh!

2010YOUY01 Apr 14, 2025

Choose a reason for hiding this comment

Uh oh!

alamb commented Apr 15, 2025

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

alamb commented Apr 16, 2025

Uh oh!

alamb commented Apr 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

alamb commented Apr 17, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

kumarlokesh commented Apr 5, 2025 •

edited

Loading

alamb commented Apr 16, 2025 •

edited

Loading