[C++] Add option to CSV reader to dictionary encode individual columns or all string / binary columns

[asfimport]

For many datasets, dictionary encoding everything can result in drastically lower memory usage and subsequently better performance in doing analytics

One difficulty of dictionary encoding in multithreaded conversions is that ideally you end up with one dictionary at the end. So you have two options:

• Implement a concurrent hashing scheme – for low cardinality dictionaries, the overhead associated with mutex contention will not be meaningful, for high cardinality it can be more of a problem

• Hash each chunk separately, then normalize at the end

  My guess is that a crude concurrent hash table with a mutex to protect mutations and resizes is going to outperform the latter

Reporter: Wes McKinney / @wesm
Assignee: Antoine Pitrou / @pitrou

Related issues:

• [C++] Add an incomplete dictionary type (depends upon)

PRs and other links:

• GitHub Pull Request #5785

_{Note: This issue was originally created as ARROW-3408. Please see the migration documentation for further details.}

[asfimport]

Wes McKinney / @wesm:
@pitrou if this makes the cut for 0.12 that's great, but no rush on it

[asfimport]

Antoine Pitrou / @pitrou:
@wesm Are chunked dictionary arrays still supposed to have the same dictionary for all chunks?

[asfimport]

Wes McKinney / @wesm:
Nope. That's what ARROW-3144 was about, so each chunk can have a different dictionary

[asfimport]

Wes McKinney / @wesm:
Issue resolved by pull request 5785
#5785