Skip to content

Conversation

@maxburke
Copy link
Contributor

Which issue does this PR close?

Closes #3910

Rationale for this change

To be able to use FixedSizeBinary for sorting / grouping, we need to implement a ScalarValue variant for this type.

@maxburke
Copy link
Contributor Author

This is dependent on apache/arrow-rs#2905

@alamb alamb added the waiting-on-upstream PR is waiting on an upstream dependency to be updated label Oct 21, 2022
Copy link
Contributor

@alamb alamb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @maxburke -- this code in this PR looks great to me.

You mentioned it depends on #3910 but it seems to pass all the tests. I wonder what depends on #3910?

Also, could you please add some sort of basic test showing this type being used ? Perhaps a SQL or dataframe test?

Value::LargeBinaryValue(s.to_owned())
})
}
scalar::ScalarValue::FixedSizeBinary(_, _) => todo!(),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you please return a "NotImplemented" error here rather than a todo() panic?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. The error type in to_proto doesn't have the NotImplented variant as it isn't a DataFusionError, hopefully a string saying that it isn't implemented is sufficient?

@github-actions github-actions bot added the core Core DataFusion crate label Oct 22, 2022
@maxburke
Copy link
Contributor Author

Thanks @maxburke -- this code in this PR looks great to me.

You mentioned it depends on #3910 but it seems to pass all the tests. I wonder what depends on #3910?

Also, could you please add some sort of basic test showing this type being used ? Perhaps a SQL or dataframe test?

I added some tests.

I may have been mistaking the dependency on arrow-rs/2905 for some other thing that was exploding at the same time in our code while implementing this. It does seem like basic ordering and grouping operations work without it.

Copy link
Contributor

@alamb alamb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great -- thanks @maxburke

}

#[tokio::test]
async fn fixed_size_binary_columns() {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

Value::LargeBinaryValue(s.to_owned())
})
}
scalar::ScalarValue::FixedSizeBinary(_, _) => {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Filed #3928 to track

Dandandan pushed a commit to yuuch/arrow-datafusion that referenced this pull request Nov 5, 2022
* implement ScalarValue for FixedSizeBinary

* Return an error instead of panicking

* add tests for scalar value fixedsizebinary variant
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

core Core DataFusion crate waiting-on-upstream PR is waiting on an upstream dependency to be updated

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ScalarValue not implemented for FixedSizeBinary types

2 participants