[C++][Dataset] Implement InMemoryDatasetFactory

[asfimport]

This will allow in memory datasets (such as tables) to participate in discovery through UnionDatasetFactory. This class will be trivial since Inspect will do nothing but return the table's schema, but is necessary to ensure that the resulting UnionDataset's unified schema accommodates the table's schema (for example including fields present only in the table's schema or emitting an error when unification is not possible)

Reporter: Ben Kietzman / @bkietz
Assignee: Ben Kietzman / @bkietz

Related issues:

• [C++][Dataset] Let datasets be viewable with non-identical schema (is superceded by)

_{Note: This issue was originally created as ARROW-7945. Please see the migration documentation for further details.}

[asfimport]

Krisztian Szucs / @kszucs:
We also should have convenience factories for InMemoryDataset, see discussion https://github.com/apache/arrow/pull/6470/files/881ee8538a2cd0c3173feb9c370b5da69388b5f4#r384503399

InMemoryDataset::Make(Table);
InMemoryDataset::Make(std::vector<RecordBatch>);  // all record batches should have the same schema

[asfimport]

Niklas B:
Would this also make us able (in the end) to create a dataset from a RecordBatch or Table? Specifically: 

    # TODO(kszucs): support InMemoryDataset for a table input

From https://github.com/apache/arrow/blob/master/python/pyarrow/dataset.py#L660

That would be awesome :) that would (if my understand is correct) open up for using filtering on Table/RecordBatch similar to what you can do with the new dataset API for parquet files, but for in memory datasets. 

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
[~bivald] that would indeed enable that, but not that you can already filter Tables/RecordBatches right now (only a different syntax, with the filter() method expecting a boolean mask)

[asfimport]

Niklas B:
@jorisvandenbossche  That's a good point! I have tried using numpy to create a boolean mask and while that operations is zero copy and lightning fast, the actual filter takes a fair bit of time (2 seconds) to filter out about 50% of 700k rows (~200 columns). I'm guessing its iterating each column, where each column takes about 0.01s. Possibly I can improve this if I write a function in cython/c++, but it's slightly out of my abilities. I'll probably wait for the Dataset API :) 

[asfimport]

Ben Kietzman / @bkietz:
Since datasets' schemas can be unified after construction, it no longer makes sense to do this unification in discovery.

[asfimport]

Niklas B:
@bkietz  is there any code samples on how to do that? Looking to use the new dataset api.