Skip to content

chore: Upgrade rand crate and some other minor crates#14967

Closed
comphead wants to merge 0 commit intoapache:mainfrom
comphead:dev
Closed

chore: Upgrade rand crate and some other minor crates#14967
comphead wants to merge 0 commit intoapache:mainfrom
comphead:dev

Conversation

@comphead
Copy link
Contributor

@comphead comphead commented Mar 3, 2025

Which issue does this PR close?

  • Closes #.

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

@github-actions github-actions bot added physical-expr Changes to the physical-expr crates core Core DataFusion crate common Related to common crate execution Related to the execution crate proto Related to proto crate functions Changes to functions implementation datasource Changes to the datasource crate labels Mar 3, 2025
@comphead comphead marked this pull request as draft March 3, 2025 15:52
@comphead
Copy link
Contributor Author

comphead commented Mar 3, 2025

Looks like some tests relying on random data generation may fail if the rand version changed

@comphead comphead force-pushed the dev branch 4 times, most recently from b6ef148 to 6f3b6c5 Compare March 12, 2025 15:37
@comphead
Copy link
Contributor Author

comphead commented Mar 14, 2025

Depends on apache/arrow-rs#7293

@comphead
Copy link
Contributor Author

comphead commented Mar 14, 2025

Depends on apache/arrow-rs#7126.

@comphead
Copy link
Contributor Author

comphead commented Mar 14, 2025

wasm pack fails because getrandom 0.2.x attached to rand 0.8.x.
wasm requires js feature to be enabled, but the feature renamed from js to wasm-js in getrandom 0.3 and this incosistency in versions feature names seems cannot let wasm to compile.

@comphead
Copy link
Contributor Author

comphead commented Mar 14, 2025

@mbrobbel cc as you working on the migration of arrow-rs

comphead
comphead commented Mar 14, 2025
View reviewed changes
datafusion/core/tests/parquet/filter_pushdown.rs Outdated
#[tokio::test]
// The result can be flaky if `rand` crate updated.
//
// Reason: Data generated by
Copy link
Contributor Author

@comphead comphead Mar 14, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These fuzzy tests based random data gen become non relevant because of rand version upgrade and seems the seed now returns different random values, different from what tests expect in terms of data distribution, column data, etc making the test useless. Considering the Filter Pushdown is important test and conditions in the tests are carefully chosen to cover use cases, we probably materialize the parquet file and get rid of random data gen to avoid test to be stale when rand crate upgraded.

@mbrobbel WDYT?

Copy link
Member

@mbrobbel mbrobbel Mar 17, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I noticed the same (in #14447) for some different tests, which I fixed by explicitly using rand_chacha: e.g. https://github.com/apache/datafusion/pull/14447/files#diff-4a599584dfc900ec21169f4f820a1b1db46b004b77533dab83a6178d5d3a467eR4253 following https://rust-random.github.io/book/crate-reprod.html.

@comphead
Copy link
Contributor Author

comphead commented Mar 26, 2025

Depends on apache/arrow-rs#7084

@comphead comphead mentioned this pull request Mar 28, 2025
Closed
@comphead
Copy link
Contributor Author

comphead commented Apr 6, 2025

this wasm PR might be related https://github.com/apache/datafusion/pull/15595/files

@github-actions github-actions bot added the catalog Related to the catalog crate label Apr 13, 2025
@comphead comphead closed this May 15, 2025
@comphead
Copy link
Contributor Author

comphead commented May 15, 2025

oops

@comphead comphead mentioned this pull request May 15, 2025
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer

@mbrobbel mbrobbel mbrobbel left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

catalog Related to the catalog crate common Related to common crate core Core DataFusion crate datasource Changes to the datasource crate execution Related to the execution crate functions Changes to functions implementation physical-expr Changes to the physical-expr crates proto Related to proto crate

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@comphead @mbrobbel