ARROW-11626: [Rust][DataFusion] Move [DataFusion] examples to own project #9494

Dandandan · 2021-02-14T16:17:49Z

Same idea as #9493 but for examples in DataFusion

Clean + building with cargo test.

Moving the micro benchmarks out of the crate is also another possibility.

	Old	New
Nr dependencies	309	249
build time(s)	77	68

github-actions · 2021-02-14T16:18:07Z

https://issues.apache.org/jira/browse/ARROW-11626

codecov-io · 2021-02-14T19:11:20Z

Codecov Report

Merging #9494 (edbea69) into master (bfa99d9) will decrease coverage by 0.07%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #9494      +/-   ##
==========================================
- Coverage   82.49%   82.42%   -0.08%     
==========================================
  Files         245      245              
  Lines       57347    57646     +299     
==========================================
+ Hits        47311    47517     +206     
- Misses      10036    10129      +93

Impacted Files	Coverage Δ
rust/datafusion-examples/examples/flight_server.rs	`0.00% <ø> (ø)`
rust/datafusion-examples/examples/simple_udaf.rs	`0.00% <ø> (ø)`
rust/parquet/src/basic.rs	`87.22% <0.00%> (-10.04%)`	⬇️
rust/parquet/src/arrow/array_reader.rs	`77.53% <0.00%> (-0.08%)`	⬇️
rust/parquet/src/schema/parser.rs	`90.20% <0.00%> (ø)`
rust/parquet/src/file/footer.rs	`96.26% <0.00%> (+0.03%)`	⬆️
rust/parquet/src/file/writer.rs	`95.32% <0.00%> (+0.21%)`	⬆️
rust/parquet/src/schema/types.rs	`89.79% <0.00%> (+0.25%)`	⬆️
rust/parquet/src/file/metadata.rs	`92.96% <0.00%> (+0.74%)`	⬆️
rust/arrow/src/array/equal/utils.rs	`76.47% <0.00%> (+0.98%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update bfa99d9...edbea69. Read the comment docs.

alamb · 2021-02-15T11:21:09Z

This is a cool idea @Dandandan -- what do you think about doing this via features rather than its own crate? I can't really articulate clearly why I would prefer one way over the other. I will try and get my thoughts formed more coherently later today

Dandandan · 2021-02-15T22:39:46Z

I have the feeling that using more features rather than moving the examples would be adding more knobs / complexity to the DataFusion crate. But I'm open for both options.

alamb · 2021-03-03T11:33:09Z

@Dandandan I think this idea is worth pursuing -- would you be willing to ressurect this PR?

Dandandan · 2021-03-04T18:10:49Z

PR is resurrected! (I think) @alamb

alamb

I like this. What do you think @andygrove and @jorgecarleitao ?

alamb · 2021-03-04T19:55:52Z

rust/datafusion-examples/Cargo.toml

FYI @andygrove -- I am not sure what you think about possibly releasing the examples as a crate of their own. I personally think it would be necessary. However perhaps it would be the right way to to eventually to somehow link to the example code from the docs.rs page https://docs.rs/datafusion/3.0.0/datafusion/

I filed this JIRA https://issues.apache.org/jira/browse/ARROW-11863 to track that

I'd lean on not publishing by default, so I'd prefer a publish = false on the Cargo file. I support clarifying this as part of ARROW-11863 @alamb.

Added publish = false.
The examples are currently also not on https://docs.rs/datafusion/3.0.0/datafusion/ I think - agree that it would be very useful.

The ok, outcome would be to point the docs from docs.rs always at master

another (better) option would be to link at a versioned url (based on published version)
https://github.com/apache/arrow/tree/apache-arrow-3.0.0

alamb · 2021-03-04T19:56:08Z

rust/datafusion/Cargo.toml

nevi-me · 2021-03-22T17:07:50Z

@alamb @Dandandan are we still planning on completing this? The benefits are tangible enough as they reduce the number of required dependencies, and we'll catch any breakages with CI###

Dandandan · 2021-03-22T17:11:37Z

@nevi-me yeah - that's the idea from my side at least. I will update the PR again so it could be merged if it has enough support

nevi-me

A change to not publish this, otherwise I think we can go ahead with this PR

nevi-me · 2021-03-22T17:15:50Z

rust/datafusion-examples/Cargo.toml

I'd lean on not publishing by default, so I'd prefer a publish = false on the Cargo file. I support clarifying this as part of ARROW-11863 @alamb.

rust/datafusion-examples/Cargo.toml

nevi-me

LGTM, pending CI

alamb · 2021-03-23T17:41:54Z

Looks like test failed with

error: Package `datafusion-examples v4.0.0-SNAPSHOT (/__w/arrow/arrow/rust/datafusion-examples)` does not have the feature `cli`

😞

alamb

I think we just need to clean up the CI and then this will be good to go. Thank you so much @Dandandan !

.github/workflows/rust.yml

alamb · 2021-03-24T15:47:20Z

Oh no ! this PR seems like it needs a rebase

Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

alamb · 2021-03-28T16:48:17Z

I have rebased this PR against master and hopefully will get it merged once CI passes

Dandandan · 2021-03-28T16:55:56Z

thanks a lot @alamb !

alamb · 2021-03-28T17:26:53Z

The integration test failure seems unrelated (out of space): https://github.com/apache/arrow/pull/9494/checks?check_run_id=2213195937


+ source_dir=/arrow/js
+ with_docs=/build
+ pushd /arrow/js
+ npm install
/arrow/js /
npm WARN tar ENOSPC: no space left on device, write
npm WARN tar ENOSPC: no space left on device, write
npm ERR! cb() never called!

alamb · 2021-03-28T17:26:58Z

I am going to merge this one

github-actions bot added Component: Rust Component: Rust - DataFusion labels Feb 14, 2021

Dandandan mentioned this pull request Feb 14, 2021

ARROW-11298: [Rust][DataFusion] Implement Postgres String Functions [Splitting to separate PRs] #9243

Closed

Dandandan changed the title ~~ARROW-11626: [Rust][DataFusion] Move datafusion examples to own project~~ ARROW-11626: [Rust][DataFusion] Move [DataFusion] examples to own project Feb 14, 2021

alamb approved these changes Mar 4, 2021

View reviewed changes

Dandandan force-pushed the move_datafusion_examples branch from f7f6103 to edbea69 Compare March 7, 2021 21:43

nevi-me requested changes Mar 22, 2021

View reviewed changes

nevi-me approved these changes Mar 22, 2021

View reviewed changes

alamb approved these changes Mar 23, 2021

View reviewed changes

.github/workflows/rust.yml Outdated Show resolved Hide resolved

alamb added the needs-rebase A PR that needs to be rebased by the author label Mar 24, 2021

Dandandan and others added 10 commits March 28, 2021 12:42

Move datafusion examples to own project

da49546

Remove dependencies

73235ba

Fix CI

225a684

Add to prepare-test

cebc63d

Move arrow-flight

f776ce9

Use latest version

ac2f2ee

Add to both tests

4626305

Fix

94dadb0

Add publish=false

835bc47

Update .github/workflows/rust.yml

19884c3

Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

alamb force-pushed the move_datafusion_examples branch from bd26b02 to 19884c3 Compare March 28, 2021 16:47

alamb closed this in 2661fc3 Mar 28, 2021

asfimport mentioned this pull request Mar 28, 2021

[Rust][DataFusion] Move DataFusion examples to own project to reduce nr dependencies #27490

Closed

ARROW-11626: [Rust][DataFusion] Move [DataFusion] examples to own project #9494

ARROW-11626: [Rust][DataFusion] Move [DataFusion] examples to own project #9494

Uh oh!

Conversation

Dandandan commented Feb 14, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Feb 14, 2021

Uh oh!

codecov-io commented Feb 14, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

alamb commented Feb 15, 2021

Uh oh!

Dandandan commented Feb 15, 2021

Uh oh!

alamb commented Mar 3, 2021

Uh oh!

Dandandan commented Mar 4, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

alamb Mar 4, 2021

Choose a reason for hiding this comment

Uh oh!

nevi-me Mar 22, 2021

Choose a reason for hiding this comment

Uh oh!

Dandandan Mar 22, 2021

Choose a reason for hiding this comment

Uh oh!

Dandandan Mar 22, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alamb Mar 4, 2021

Choose a reason for hiding this comment

Uh oh!

nevi-me commented Mar 22, 2021

Uh oh!

Dandandan commented Mar 22, 2021

Uh oh!

nevi-me left a comment

Choose a reason for hiding this comment

Uh oh!

nevi-me Mar 22, 2021

Choose a reason for hiding this comment

Uh oh!

Uh oh!

nevi-me left a comment

Choose a reason for hiding this comment

Uh oh!

alamb commented Mar 23, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

alamb commented Mar 24, 2021

Uh oh!

alamb commented Mar 28, 2021

Uh oh!

Dandandan commented Mar 28, 2021

Uh oh!

alamb commented Mar 28, 2021

Uh oh!

alamb commented Mar 28, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Dandandan commented Feb 14, 2021 •

edited

Loading

codecov-io commented Feb 14, 2021 •

edited

Loading

Dandandan commented Mar 4, 2021 •

edited

Loading

Dandandan Mar 22, 2021 •

edited

Loading

alamb commented Mar 23, 2021 •

edited

Loading