[GLUTEN-9335][VL] Support iceberg partition write#10497
[GLUTEN-9335][VL] Support iceberg partition write#10497jinchengchenghh merged 10 commits intoapache:mainfrom
Conversation
|
Run Gluten Clickhouse CI on x86 |
jinchengchenghh
force-pushed
the
iceberg-partition
branch
from
76fff49 to
a8464e0
Compare
|
Run Gluten Clickhouse CI on x86 |
jinchengchenghh
force-pushed
the
iceberg-partition
branch
from
a8464e0 to
77ebf8a
Compare
|
Run Gluten Clickhouse CI on x86 |
1 similar comment
|
Run Gluten Clickhouse CI on x86 |
jinchengchenghh
force-pushed
the
iceberg-partition
branch
from
04c758e to
8c5a345
Compare
|
Run Gluten Clickhouse CI on x86 |
1 similar comment
|
Run Gluten Clickhouse CI on x86 |
jinchengchenghh
force-pushed
the
iceberg-partition
branch
from
66ed868 to
7165e15
Compare
|
Run Gluten Clickhouse CI on x86 |
5 similar comments
|
Run Gluten Clickhouse CI on x86 |
|
Run Gluten Clickhouse CI on x86 |
|
Run Gluten Clickhouse CI on x86 |
|
Run Gluten Clickhouse CI on x86 |
|
Run Gluten Clickhouse CI on x86 |
jinchengchenghh
force-pushed
the
iceberg-partition
branch
from
ceace8d to
91cd5ed
Compare
|
Run Gluten Clickhouse CI on x86 |
jinchengchenghh
force-pushed
the
iceberg-partition
branch
from
91cd5ed to
a483618
Compare
|
Run Gluten Clickhouse CI on x86 |
| auto veloxCfg = | ||
| std::make_shared<facebook::velox::config::ConfigBase>(std::unordered_map<std::string, std::string>(sparkConfs)); | ||
| auto connectorSessionProperties_ = getHiveConfig(veloxCfg); | ||
| connectorSessionProperties_->set( |
There was a problem hiding this comment.
May need to do a refactor with WholeStageResultIterator::createConnectorConfig
jinchengchenghh
force-pushed
the
iceberg-partition
branch
from
a483618 to
5b8eb7b
Compare
|
Run Gluten Clickhouse CI on x86 |
1 similar comment
|
Run Gluten Clickhouse CI on x86 |
jinchengchenghh
force-pushed
the
iceberg-partition
branch
from
a3be695 to
651d2ea
Compare
|
Run Gluten Clickhouse CI on x86 |
zhouyuan
left a comment
zhouyuan
left a comment
There was a problem hiding this comment.
@jinchengchenghh can you please do a rebase?
cpp/velox/utils/ConfigExtractor.cc
Outdated
| hiveConfMap[facebook::velox::connector::hive::HiveConfig::kOrcUseColumnNames] = "true"; | ||
|
|
||
| return std::make_shared<facebook::velox::config::ConfigBase>(std::move(hiveConfMap)); | ||
| return std::make_shared<facebook::velox::config::ConfigBase>(std::move(hiveConfMap), true); |
There was a problem hiding this comment.
This config may introduce issue on latest main, as the azure client is expects some readonly config
There was a problem hiding this comment.
|
Run Gluten Clickhouse CI on x86 |
jinchengchenghh
force-pushed
the
iceberg-partition
branch
from
bca9ee2 to
4dfcb19
Compare
|
Run Gluten Clickhouse CI on x86 |
|
Run Gluten Clickhouse CI on x86 |
|
Run Gluten Clickhouse CI on x86 |
|
Could you help review again? Thanks! @zhouyuan |
zhouyuan
left a comment
zhouyuan
left a comment
There was a problem hiding this comment.
👍
could we also create a sub-issue for the remaining fanout fix?
|
Yes, added #10617 |
2 participants
Add Protobuf struct IcebergPartitionField to transfer the iceberg id information, add IcebergPartitionSpec to transfer partition information.
Build with test and benchmark in CI and fix IcebergWriteTest build.
Set the file format to orc to bypass native parquet write for partitioned tpch iceberg suite, after facebookincubator/velox#14670 which supports fanout false mode merged, we can relax the restriction.
Relevant PR: facebookincubator/velox#13874