Skip to content

Consolidate feature flags into configuration guide #14657

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
alamb wants to merge 7 commits into from
Closed

Consolidate feature flags into configuration guide #14657

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
57e0f39
Move crate configuration to other configuration information
alamb Feb 13, 2025
e60db0c
tweal
alamb Feb 13, 2025
705e180
fix toml
alamb Feb 13, 2025
fbcece9
Merge remote-tracking branch 'apache/main' into alamb/feature_flags
alamb Feb 22, 2025
df190ad
Fix: add correct feature flags
alamb Feb 22, 2025
6654168
alphabetize options
alamb Feb 22, 2025
49b8cf5
Merge branch 'main' into alamb/feature_flags
alamb May 5, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 2 additions & 25 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -106,32 +106,9 @@ Please see the [contributor guide] and [communication] pages for more informatio

## Crate features

This crate has several [features] which can be specified in your `Cargo.toml`.
Please see the [Crate features section of the user guide] for available features

[features]: https://doc.rust-lang.org/cargo/reference/features.html

Default features:

- `nested_expressions`: functions for working with nested type function such as `array_to_string`
- `compression`: reading files compressed with `xz2`, `bzip2`, `flate2`, and `zstd`
- `crypto_expressions`: cryptographic functions such as `md5` and `sha256`
- `datetime_expressions`: date and time functions such as `to_timestamp`
- `encoding_expressions`: `encode` and `decode` functions
- `parquet`: support for reading the [Apache Parquet] format
- `regex_expressions`: regular expression functions, such as `regexp_match`
- `unicode_expressions`: Include unicode aware functions such as `character_length`
- `unparser`: enables support to reverse LogicalPlans back into SQL
- `recursive_protection`: uses [recursive](https://docs.rs/recursive/latest/recursive/) for stack overflow protection.

Optional features:

- `avro`: support for reading the [Apache Avro] format
- `backtrace`: include backtrace information in error messages
- `pyarrow`: conversions between PyArrow and DataFusion types
- `serde`: enable arrow-schema's `serde` feature

[apache avro]: https://avro.apache.org/
[apache parquet]: https://parquet.apache.org/
[crate features section of the user guide]: https://datafusion.apache.org/user-guide/crate-configuration.html#crate-features

## Rust Version Compatibility Policy

Expand Down
3 changes: 3 additions & 0 deletions datafusion/core/Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,8 +57,10 @@ default = [
"datetime_expressions",
"encoding_expressions",
"regex_expressions",
"math_expressions",
"string_expressions",
"unicode_expressions",
"unparser",
"compression",
"parquet",
"recursive_protection",
Expand Down Expand Up @@ -86,6 +88,7 @@ serde = [
"arrow-schema/serde",
]
string_expressions = ["datafusion-functions/string_expressions"]
unparser = ["datafusion-sql/unparser"]
unicode_expressions = [
"datafusion-sql/unicode_expressions",
"datafusion-functions/unicode_expressions",
Expand Down
42 changes: 41 additions & 1 deletion docs/source/user-guide/crate-configuration.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,47 @@ control DataFusion's behavior.

[configuration settings]: configs.md

## Add latest non published DataFusion dependency
## Crate features

The `datafusion` crate has several [features] which can be specified in your `Cargo.toml`. For example
to enable support for `avro` you can put the following in your `Cargo.toml`:

```toml
datafusion = { version = "45", features = ["avro"] }
```

[features]: https://doc.rust-lang.org/cargo/reference/features.html

You can always check the [Cargo.toml] file for the most up to date list of features.

[cargo.toml]: https://github.com/apache/datafusion/blob/main/datafusion/core/Cargo.toml#L39

### Default features:

- `compression`: reading files compressed with `xz2`, `bzip2`, `flate2`, and `zstd`
- `crypto_expressions`: cryptographic functions such as `md5` and `sha256`
- `datetime_expressions`: date and time functions such as `to_timestamp`
- `encoding_expressions`: `encode` and `decode` functions
Copy link
Contributor Author

@alamb alamb Feb 13, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There were also a few missing items here

- `math_expressions`: `sqrt` and other math functions
Copy link
Contributor

@dhegberg dhegberg Feb 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This doesn't appear to be in the defaults list here:

datafusion/datafusion/core/Cargo.toml

Line 49 in 40bb75f

default = [

Copy link
Contributor Author

@alamb alamb Feb 22, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That was an excellent call. I fixed it in df190ad

However, I think since the subcrates already have their own default features, it is likely not possible to remove these various functon packages 🤔 maybe we should fix that in a following issue

- `nested_expressions`: functions for working with nested type function such as `array_to_string`
- `parquet`: support for reading the [Apache Parquet] format
- `string_expressions`: string functions such as `concat`
- `recursive_protection`: uses [recursive](https://docs.rs/recursive/latest/recursive/) for stack overflow protection.
- `regex_expressions`: regular expression functions, such as `regexp_match`
- `unicode_expressions`: Include unicode aware functions such as `character_length`
- `unparser`: enables support to reverse LogicalPlans back into SQL
Copy link
Contributor

@dhegberg dhegberg Feb 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This doesn't appear to be in the defaults list here:

datafusion/datafusion/core/Cargo.toml

Line 49 in 40bb75f

default = [


### Optional features:

- `avro`: support for reading the [Apache Avro] format
- `backtrace`: include backtrace information in error messages
- `pyarrow`: conversions between PyArrow and DataFusion types
- `serde`: enable arrow-schema's `serde` feature

[apache avro]: https://avro.apache.org/
[apache parquet]: https://parquet.apache.org/

## Use latest non published DataFusion dependency

DataFusion changes are published to `crates.io` according to the [release schedule](https://github.com/apache/datafusion/blob/main/dev/release/README.md#release-process)

Expand Down