Skip to content

[WIP] Fix E2E DataFrameTests for Spark 3.0. #365

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
elvaliuliuliu wants to merge 1 commit into from
Closed

[WIP] Fix E2E DataFrameTests for Spark 3.0. #365

elvaliuliuliu wants to merge 1 commit into from

Conversation

@elvaliuliuliu
Copy link
Contributor

@elvaliuliuliu elvaliuliuliu commented Dec 5, 2019

This fixes E2E DataFrameTests that are failing for Spark 3.0. because the APIs are either removed or deprecated.

This is a part of the effort to bring in CI for Spark 3.0: #348.

@elvaliuliuliu
Copy link
Contributor Author

elvaliuliuliu commented Dec 5, 2019

This PR currently fixes TestSignaturesV2_3_X.

Moreover, not only TestGroupedMapUdf is failing but also TestVectorUdf, TestWithColumn, TestUdf in DataFrameTests for Spark 3.0. They are all failing because of collectToPython API.

But I am not sure how to fix DataFrame.collect() which is using collectToPython. @imback82 / @suhsteve Any suggestions? Thanks!

imback82
imback82 reviewed Dec 6, 2019
View reviewed changes
src/csharp/Microsoft.Spark/Sql/DataFrame.cs
/// This API is removed in Spark 3.0.
/// </remarks>
/// <returns>Row objects</returns>
[Removed(Versions.V3_0_0)]
Copy link
Contributor

@imback82 imback82 Dec 6, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not removed. They changed the contract. We should follow this: https://github.com/apache/spark/blob/0a7095156bdb565133f7dcc74546c51a5e5d2414/python/pyspark/rdd.py#L175

@imback82
Copy link
Contributor

imback82 commented Dec 6, 2019

Moreover, not only TestGroupedMapUdf is failing but also TestVectorUdf, TestWithColumn, TestUdf in DataFrameTests for Spark 3.0. They are all failing because of collectToPython API.

Did you build the worker with the latest changes? I think I fixed it in #311.

@elvaliuliuliu
Copy link
Contributor Author

elvaliuliuliu commented Dec 6, 2019

Moreover, not only TestGroupedMapUdf is failing but also TestVectorUdf, TestWithColumn, TestUdf in DataFrameTests for Spark 3.0. They are all failing because of collectToPython API.

Did you build the worker with the latest changes? I think I fixed it in #311.

It works for me after I updated worker. Thanks!

@elvaliuliuliu elvaliuliuliu changed the title Fix E2E DataFrameTests for Spark 3.0. [WIP] Fix E2E DataFrameTests for Spark 3.0. Dec 6, 2019
@imback82
Copy link
Contributor

imback82 commented Sep 5, 2020

This is being addressed by #655.

@imback82 imback82 closed this Sep 5, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer

@imback82 imback82 imback82 left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@elvaliuliuliu @imback82