[Rust] [DataFusion] Add support for is_in

[asfimport]

Add support for a function that returns true if an element belongs to a finite list of elements in a list.

The list should be an arrow array (to be dynamically-typed).

 

Reporter: Jorge Leitão / @jorgecarleitao
Assignee: Mike Seddon / @seddonm1

Related issues:

• [Rust] [DataFusion] Improve performance if IN list function (is related to)

PRs and other links:

• GitHub Pull Request #9038

_{Note: This issue was originally created as ARROW-10356. Please see the migration documentation for further details.}

[asfimport]

Neville Dipale / @nevi-me:
I had some list kernels at https://github.com/nevi-me/rust-dataframe/blob/master/src/functions/array.rs. The code is old, so it's written with the builder pattern, but some of it could be used as a start.

@jorgecarleitao  how about we classify kernels into groups, and implement them as subtasks in an umbrella JIRA? It might help with completeness for say list kernels. We could use Spark as inspiration for what to support (as I had done above)

[asfimport]

Vivek Akupatni:
Hi all, I would like to work on it. But I am not sure where to start. Any pointers/references or what is expected would be useful for me.
Thanks for your time.

[asfimport]

Jorge Leitão / @jorgecarleitao:
Hi [~apatniv], Sure!

So, broad steps, we can split this issue in two:

• changes in the arrow crate

• changes in the datafusion crate

  Arrow:

1. fork the repo on github and clone it to your computer

2. bootsrap your working environment (e.g. vs code + git + rust analyzer plugin)

3. create a new module is_in.rs on the arrow crate under compute/kernels.

4. Design the function. E.g. is_in is something like fn is_in(array1, array2) -> boolean array.

   1. Can we leverage the arrow format somehow, or do we need to do it directly on the values? E.g. is_in seems to be a NxM problem, as we need to check each element in array1 on array2. Is there any way to speed this process?
   2. which utility methods do we may need?
   3. Any edge cases?

5. Implement. Check other kernels for inspiration. It is fine to PR only a subset of all types. e.g. start with the implementation for only primitive arrays (int32, int64, etc.)

6. Add tests (can be done first)

7. Update README

8. PR (optional, good to get initial feedback :))

   DataFusion:

9. Use the kernel defined above on data fusion on physical_plan/functions.rs

10. Add the function is_in to the impl of Expr in logical_plan.rs

11. Update README

12. PR

    Let us know if you have any questions!

[asfimport]

Neville Dipale / @nevi-me:
@jorgecarleitao  is this not the same as ARROW-10103? We called it 'contains' there, where we check if a list contains a primitive, returning a boolean. If it is, perhaps we could use this to expand the datatypes supported there (if we don't support all), and maybe tweak behaviour where desired (there were some questions around null behaviour at the time)

[asfimport]

Vivek Akupatni:
Just to confirm are these the corresponding code repos.
https://github.com/apache/arrow/tree/master/rust/arrow
https://github.com/apache/arrow/tree/master/rust/datafusion

[asfimport]

Jorge Leitão / @jorgecarleitao:
[~nevi_me], it is. I think we should move that to compute::kernels, for consistency.

 

[~apatniv], I am really sorry, this is a duplicate, I did not know this was already supported. Yes, those are the directories.

EDIT: this is not supported in DataFusion, AFAIK. So, there is still work in this issue to add it to DataFusion.

[asfimport]

Vivek Akupatni:
[~nevi_me] Not a problem. I can work on integrating into DataFusion.

[asfimport]

Andrew Lamb / @alamb:
Issue resolved by pull request 9038
#9038

[asfimport]

Andrew Lamb / @alamb:
FWIW @seddonm1 has implemented this – there is still other work that is possible – see https://issues.apache.org/jira/browse/ARROW-11182