[R] Add chunk_size to Table$create()

[asfimport]

While working on ARROW-3308, I noticed that write_feather has a chunk_size argument, which by default will write batches of 64k rows into the file. In principle, a chunking strategy like this would prevent the need to bump up to large_utf8 when ingesting a large character vector because you'd end up with many chunks that each fit into a regular utf8 type. However, the way the function works, the data.frame is converted to a Table with all ChunkedArrays containing a single chunk first, which is where the large_utf8 type gets set. But if Table$create() could be instructed to make multiple chunks, this would be resolved.

Reporter: Neal Richardson / @nealrichardson

Related issues:

• [Python][R] Expose incremental write API for Feather files (is related to)
• [R] Use Converter API to convert SEXP to Array/ChunkedArray (is related to)

_{Note: This issue was originally created as ARROW-9293. Please see the migration documentation for further details.}

[asfimport]

Wes McKinney / @wesm:
Makes sense to me

[asfimport]

Romain Francois / @romainfrancois:
Assuming this comes after #8650 it boils down to vec_to_arrow() accepting some sort of chunking policy, and in turn means that the converter api needs something too (this is essentially https://issues.apache.org/jira/browse/ARROW-5628)

The api we use now in the converter api goes through: 

    Status Extend(SEXP x, int64_t size) override;

which means ingest x and btw it has this number of elements. We need some way to be able to express "ingest that range of elements from x". The Chunker class, at least in its current form does not help. 

[asfimport]

Antoine Pitrou / @pitrou:
@romainfrancois Do you still plan to work on this?

[asfimport]

Antoine Pitrou / @pitrou:
cc @thisisnic

[asfimport]

Todd Farmer / @toddfarmer:
This issue was last updated over 90 days ago, which may be an indication it is no longer being actively worked. To better reflect the current state, the issue is being unassigned. Please feel free to re-take assignment of the issue if it is being actively worked, or if you plan to start that work soon.

[thisisnic]

This issue hasn't had activity in a long time. If it's still being worked on, please leave a comment. Otherwise, it will be closed on 23rd June.

Labelled Status: Stale-Warning for tracking.