Skip to content

ARROW-7085: Custom column builder for CSV reader #5880

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
fexolm wants to merge 1 commit into from
Closed

ARROW-7085: Custom column builder for CSV reader #5880

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 13 additions & 1 deletion cpp/src/arrow/csv/options.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,11 +19,14 @@
#define ARROW_CSV_OPTIONS_H

#include <cstdint>
#include <functional>
#include <memory>
#include <string>
#include <unordered_map>
#include <vector>

#include "arrow/csv/column_builder.h"
#include "arrow/util/task_group.h"
#include "arrow/util/visibility.h"

namespace arrow {
Expand Down Expand Up @@ -67,7 +70,16 @@ struct ARROW_EXPORT ConvertOptions {
bool check_utf8 = true;
/// Optional per-column types (disabling type inference on those columns)
std::unordered_map<std::string, std::shared_ptr<DataType>> column_types;
/// Recognized spellings for null values

/// Optional per-column fabrics for custom column builders
std::unordered_map<
std::string,
std::function<Status(MemoryPool* pool, const std::shared_ptr<DataType>& type,
int32_t col_index, const ConvertOptions& options,
const std::shared_ptr<internal::TaskGroup>& task_group,
std::shared_ptr<ColumnBuilder>* out)>>
column_builder_fabrics;
// Recognized spellings for null values
std::vector<std::string> null_values;
/// Recognized spellings for boolean true values
std::vector<std::string> true_values;
Expand Down
6 changes: 6 additions & 0 deletions cpp/src/arrow/csv/reader.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -225,6 +225,12 @@ class BaseTableReader : public csv::TableReader {
if (it == convert_options_.column_types.end()) {
return ColumnBuilder::Make(pool_, col_index, convert_options_, task_group_, out);
} else {
auto builder_fabric =
convert_options_.column_builder_fabrics.find(col_name);
if (builder_fabric != convert_options_.column_builder_fabrics.end()) {
return builder_fabric->second(pool_, it->second, col_index, convert_options_,
task_group_, out);
}
return ColumnBuilder::Make(pool_, it->second, col_index, convert_options_,
task_group_, out);
}
Expand Down