[Rust] Move filter and limit code from DataFusion into compute module

[asfimport]

FilterRelation and the new LimitRelation (in ARROW-4464) contain code for filtering and limiting arrays that could now be pushed down into the compute module.

Reporter: Andy Grove / @andygrove
Assignee: Nicolas Trinquier / @ntrinquier

PRs and other links:

• GitHub Pull Request #3741

_{Note: This issue was originally created as ARROW-4605. Please see the migration documentation for further details.}

[asfimport]

Nicolas Trinquier / @ntrinquier:
@andygrove did you mean to move the functions as they are? They both do some kind of filtering on the data, what do you think of re-implementing them as a more generic function?

Some along those lines:

fn filter(a: &Array, predicate: Fn(usize) -> bool) -> Result<ArrayRef> {
...

  for i in 0..b.len() {
    if predicate(i) {
      builder.append_value(a.value(i))?
    }
  }
...
}

 

Predicates would look like this:

let limit_predicate = |index| { index < limit_value }
let filter_predicate = |index| { filter_bools.value(index) }

I do not know a/ if this pattern is very rustacean, and b/ if the abstraction is worth it (i.e. in the case of limit we would still allocate a buffer for the full size and iterate through all the elements whereas we could save space and return early).

 

[asfimport]

Nicolas Trinquier / @ntrinquier:
Maybe it's better to have 2 functions and define 2 macros, for limit and filter.

[asfimport]

Neville Dipale / @nevi-me:
Hi @ntrinquier, for filter, you might consider 
fn filter(a: &Array, predicate: &BooleanArray)
so the predicate evaluation is evaluated once, and a boolean array passed to filter each array.

[asfimport]

Nicolas Trinquier / @ntrinquier:
filter has already the signature you suggest [~nevi_me].

However for limit it is only a usize, not an array of bools, so I initially suggested to have one function to do both filter and limit, with the signature:
filter(a: &Array, predicate: Fn(usize) -> bool) -> Result
but on second though I am not sure this is worth it because of the performance implications. We might be better off using macros. What do you think?

[asfimport]

Andy Grove / @andygrove:
Issue resolved by pull request 3741
#3741