ARROW-11962: [Rust][DataFusion] Improve DataFusion docs #9701

alamb · 2021-03-14T12:17:32Z

It might be easiest to see the changes by looking at the rendered markdown here: https://github.com/alamb/arrow/tree/alamb/update_datafusion_docs/rust/datafusion

Rationale

It would be nice to market / explain DataFusion a bit more and explain what it is good for

Changes

Describe usecases for DataFusion (Add some marketing "spin"??)
Add links to other projects using DataFusion
Add links to tech talks about DataFusion
Split out developer docs into their own file
Use the cool new image that came in ARROW-11860: [Rust] [DataFusion] Add DataFusion logos #9630

github-actions · 2021-03-14T12:17:49Z

https://issues.apache.org/jira/browse/ARROW-11962

alamb · 2021-03-14T12:19:07Z

rust/datafusion/DEVELOPERS.md

@@ -0,0 +1,79 @@
+# Developer's guide


I pulled this file into its own separate file so that it didn't appear on https://crates.io/crates/datafusion

alamb · 2021-03-14T12:19:45Z

rust/datafusion/DEVELOPERS.md

+
+### Architecture Overview
+
+* (March 2021): The DataFusion architecture is described in *Query Engine Design and the Rust-Based DataFusion in Apache Arrow*: [recording](https://www.youtube.com/watch?v=K6eCAVEk4kU) (DataFusion content starts ~ 15 minutes in) and [slides](https://www.slideshare.net/influxdata/influxdb-iox-tech-talks-query-engine-design-and-the-rustbased-datafusion-in-apache-arrow-244161934)


I added links to the talks from myself and @andygrove were added -- hopefully that is not too aggrandizing, but I think the content is helpful

I agree that the links are helpful and should be kept.

I do think that since those slides and presentations did not go under the review / PR process, it may be confusing to say that they describe the architecture.

What do you think of

There is no formal document describing DataFusion's architecture yet, but the following presentations offer a good overview of its different components and how they interact together.

alamb · 2021-03-14T12:19:54Z

rust/datafusion/README.md

+* [InfluxDB IOx](https://github.com/influxdata/influxdb_iox) Time Series Database
+* [Ballista](https://github.com/ballista-compute/ballista) Distributed Compute Platform
+* [Cloudfuse Buzz](https://github.com/cloudfuse-io/buzz-rust)
+* [ROAPI](https://github.com/roapi/roapi)


delta-rs (https://github.com/delta-io/delta-rs) also uses datafusion to for querying delta tables :) for example: https://github.com/delta-io/delta-rs/blob/main/rust/tests/datafusion_test.rs

alamb · 2021-03-14T12:20:08Z

rust/datafusion/README.md

+
+* [InfluxDB IOx](https://github.com/influxdata/influxdb_iox) Time Series Database
+* [Ballista](https://github.com/ballista-compute/ballista) Distributed Compute Platform
+* [Cloudfuse Buzz](https://github.com/cloudfuse-io/buzz-rust)


fyi @rdettai

https://github.com/cube-js/cube.js could be added?

Indeed -- https://github.com/cube-js/cube.js/blob/9a4603a857c55b868fe20e8d45536d1f1188cf44/rust/cubestore/Cargo.toml
I will add to the list. Thank you

FYI @paveltiunov

alamb · 2021-03-14T12:20:14Z

rust/datafusion/README.md

+Here are some of the projects known to use DataFusion:
+
+* [InfluxDB IOx](https://github.com/influxdata/influxdb_iox) Time Series Database
+* [Ballista](https://github.com/ballista-compute/ballista) Distributed Compute Platform


fyi @andygrove

alamb · 2021-03-14T12:23:53Z

rust/datafusion/README.md

+capable of parallel execution against partitioned data sources (CSV
+and Parquet) using threads.
+
+## Use Cases


One other thing this page might benefit from is an example (e.g. look at how tokio does it https://crates.io/crates/tokio)

Perhaps we could just lift the nice example from https://docs.rs/datafusion/3.0.0/datafusion/ ?

FYI I will do so as a follow on PR: #9710

Dandandan · 2021-03-15T11:19:39Z

rust/datafusion/README.md

+ - [x] Projection push down
+ - [x] Predicate push down
+ - [x] Constant folding
+ - [x] Limit Pushdown


+Join reordering

Dandandan · 2021-03-15T11:20:20Z

rust/datafusion/README.md

 - [ ] Lists
 - [x] Subqueries
- [ ] Joins
+- [x] Joins


Inner/Left/right/ (no cross and outer joins)

jorgecarleitao

Thanks a lot for this, @alamb ; it seriously improved the appearance and selling points :)

jorgecarleitao · 2021-03-15T14:50:23Z

rust/datafusion/DEVELOPERS.md

+
+### Architecture Overview
+
+* (March 2021): The DataFusion architecture is described in *Query Engine Design and the Rust-Based DataFusion in Apache Arrow*: [recording](https://www.youtube.com/watch?v=K6eCAVEk4kU) (DataFusion content starts ~ 15 minutes in) and [slides](https://www.slideshare.net/influxdata/influxdb-iox-tech-talks-query-engine-design-and-the-rustbased-datafusion-in-apache-arrow-244161934)


I agree that the links are helpful and should be kept.

I do think that since those slides and presentations did not go under the review / PR process, it may be confusing to say that they describe the architecture.

What do you think of

There is no formal document describing DataFusion's architecture yet, but the following presentations offer a good overview of its different components and how they interact together.

jorgecarleitao · 2021-03-15T15:07:31Z

rust/datafusion/DEVELOPERS.md

+* `cargo test` to test
+* etc.
+
+### Architecture Overview


IMO we could keep this section on the README, since it concerns everyone (not just contributors / developers).

jorgecarleitao · 2021-03-15T15:10:55Z

rust/datafusion/README.md

+
+## Why DataFusion?
+
+* *High Performance*: Leveraging Rust and Arrow's memory model, DataFusion achieves superior performance


we have no benchmarks against spark, dask, pandas, etc, so we need to be careful about this claim. I think that @Dandandan was working in running benchmarks against other engines, ARROW-11252, to see it performs.

Hehe -- you caught me here selling ahead of the product. You will notice I didn't actually say what DataFusion had superior performance to. I will change this to say "achieves very high" performance without quantifying

jorgecarleitao · 2021-03-15T15:11:26Z

rust/datafusion/README.md

+* [InfluxDB IOx](https://github.com/influxdata/influxdb_iox) Time Series Database
+* [ROAPI](https://github.com/roapi/roapi)
+
+(if you know of another project, please submit a PR to add a link!)


datafusion-python? ^_^

ARROW-11962: [Rust][DataFusion] Improve DataFusion docs

d57927d

github-actions bot added Component: Rust - DataFusion Component: Rust labels Mar 14, 2021

alamb commented Mar 14, 2021

View reviewed changes

add apache license

41b7fc8

alamb requested review from andygrove and jorgecarleitao March 14, 2021 13:41

andygrove approved these changes Mar 14, 2021

View reviewed changes

houqp approved these changes Mar 14, 2021

View reviewed changes

Dandandan reviewed Mar 15, 2021

View reviewed changes

Add other projects, update feature list

3e956ec

jorgecarleitao reviewed Mar 15, 2021

View reviewed changes

alamb mentioned this pull request Mar 15, 2021

ARROW-11969: [Rust][DataFusion] Improve Examples in documentation #9710

Closed

alamb added 3 commits March 15, 2021 11:48

More improvements

4b0af44

tweak

5ef9f51

fix indent

a57ea36

alamb closed this in 23a32fb Mar 15, 2021

alamb deleted the alamb/update_datafusion_docs branch March 15, 2021 21:18

asfimport mentioned this pull request Mar 15, 2021

[Rust][DataFusion] Update Datafusion Docs / readme #27795

Closed


		### Architecture Overview

		* (March 2021): The DataFusion architecture is described in Query Engine Design and the Rust-Based DataFusion in Apache Arrow: [recording](https://www.youtube.com/watch?v=K6eCAVEk4kU) (DataFusion content starts ~ 15 minutes in) and [slides](https://www.slideshare.net/influxdata/influxdb-iox-tech-talks-query-engine-design-and-the-rustbased-datafusion-in-apache-arrow-244161934)


		## Why DataFusion?

		* High Performance: Leveraging Rust and Arrow's memory model, DataFusion achieves superior performance

ARROW-11962: [Rust][DataFusion] Improve DataFusion docs #9701

ARROW-11962: [Rust][DataFusion] Improve DataFusion docs #9701

Uh oh!

Conversation

alamb commented Mar 14, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale

Changes

Uh oh!

github-actions bot commented Mar 14, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jorgecarleitao left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alamb Mar 15, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

alamb commented Mar 14, 2021 •

edited

Loading

alamb Mar 15, 2021 •

edited

Loading