[C++] Define abstract API for filtering Arrow streams (e.g. predicate evaluation)

[asfimport]

It would be useful to be able to apply analytic predicates to an Arrow stream in a composable way. As soon as we are able to compute some simple predicates on in-memory Arrow data, we could define our first version of this

Reporter: Wes McKinney / @wesm

Related issues:

• [C++] Implement basic Filter expression classes (duplicates)

_{Note: This issue was originally created as ARROW-1036. Please see the migration documentation for further details.}

[asfimport]

Anthony Fox:
This is quite similar to a project the GeoMesa team has been working on.

The GeoMesa project has started to put together a SQL-like API over Arrow files in javascript for in-browser querying and visualization. We have defined a class called ArrowDataSet that wraps an Arrow file and exposes query and countBy/groupBy methods. The queries are defined using a set of simple predicate expressions (And, Or, Equals, LTEquals,During, etc etc) with the idea of adding spatial predicates eventually (Contains, Intersects, Overlaps). The query is received by the ArrowDataSet and a query execution plan is produced. The query execution plan has the usual operators (Scan, Filter, Project, HashGroupBy) as well as optimized Filters for dictionary encoded values. We are also planning on having a primary sort key that is hinted through the Arrow column meta-data and appropriate optimizations with additional operators like PrimarySortKeyScan. This will help with seeks when there's a predicate on the primary sort key. For instance, if the primary sort key is date and there's a query predicate using During on a start and end date, then the execution plan will use PrimarySortKeyScan to efficiently skip batches till it reaches records that pass the predicate.

I'd be interested in how this can be standardized across the languages supported by Arrow.

[asfimport]

Wes McKinney / @wesm:
This sounds really cool. Would be great to collect as much of the client-side analytics in the Arrow JS implementation. When you are ready, it would be great to have you all write a blog post for the Arrow website showing off this use csae

[asfimport]

Anthony Fox:
@wesm We'd be happy to write a blog post. We need to get it hooked up into a client-side mapping client like OpenLayers or Leaflet so there are compelling screencaps to go along with the post.

[asfimport]

Wes McKinney / @wesm:
Sure, whenever you're ready, feel free to go for it (you can make a PR into the arrow/site directory)

[asfimport]

Wes McKinney / @wesm:
Now that we can use Gandiva to generate simple predicates, it might be a good idea to implement a simple table-valued filter node

[asfimport]

Wes McKinney / @wesm:
cc @bkietz as you think about application of filters to record batches as a post-processing step

[asfimport]

Ben Kietzman / @bkietz:
@wesm I have a PR up which addresses this
https://github.com/apache/arrow/pull/5157/files#diff-157d3e1f58273646705b1760a1f09770R208

[asfimport]

Neal Richardson / @nealrichardson:
Did ARROW-6243 cover this, or is there something else to do?

[asfimport]

Francois Saint-Jacques / @fsaintjacques:
I would say yes.

[asfimport]

Todd Farmer / @toddfarmer:
Transitioning issue from Resolved to Closed to based on resolution field value.