refactor: replace custom XET sink with OpenDAL ObjectStore by davanstrien · Pull Request #2 · davanstrien/polars

davanstrien · 2026-04-02T09:57:38Z

Summary

Replace ~1,050 lines of custom XET sink code with a standard ObjectStore implementation backed by OpenDAL's HF service
HF URLs now flow through the same FileSink path as S3/GCS/Azure — no custom sink node, no IR special-case
All new logic consolidated in a single crates/polars-io/src/cloud/hf.rs module (~175 lines), following the same build_hf() pattern as build_aws()/build_gcp()/build_azure()
Rename feature flag hf_bucket_sink → hf

Motivation

Polars maintainers flagged that the custom XET code was too much HF-specific maintenance burden. The OpenDAL approach offloads XET/HF internals to OpenDAL, leaving Polars with only standard ObjectStore wiring.

What's deleted

crates/polars-io/src/cloud/hf_bucket/ (4 files, 721 lines) — custom XET upload, batch API, streaming uploader
crates/polars-stream/src/nodes/io_sinks/hf_bucket_sink.rs (260 lines) — custom sink node
IR lowering special-case, PhysNodeKind::HfBucketSink variant, graph/fmt wiring

What's added

crates/polars-io/src/cloud/hf.rs — URL parsing, token extraction, OpenDAL builder
object_store_setup.rs — 6-line CloudType::Hf match arm calling build_hf()
object_store_opendal compatibility bump from object_store 0.12 → 0.13

Dependencies

Using local path deps for opendal and object_store_opendal during development. Will swap to published crate versions once apache/opendal#7185 ships.

Test plan

cargo check -p polars-stream --features hf — compiles
cargo check -p polars-stream — no regression without feature
End-to-end test with real HF bucket (pending OpenDAL release)

🤖 Generated with Claude Code

…la-rs#26919)

…rs#26927)

…_cse` (pola-rs#26931)

…_slice_unchecked` (pola-rs#26928)

…ola-rs#26938) Co-authored-by: gabriel <gabriel.g.robin@airbus.com>

…me (pola-rs#26936)

Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

pola-rs#26946)

…26860)

Co-authored-by: moktamd <moktamd@users.noreply.github.com> Co-authored-by: nameexhaustion <simonlin.rqmmw@slmail.me>

…oes not begin with base path, or contains '..' (pola-rs#26894)

…6764) Co-authored-by: Simon Lin <simonlin.rqmmw@slmail.me>

…tring (pola-rs#26934)

Co-authored-by: Claude <noreply@anthropic.com>

…26958)

Co-authored-by: nameexhaustion <simonlin.rqmmw@slmail.me>

pola-rs#27050)

…#27011)

…ction (pola-rs#27113)

…#27104) Co-authored-by: Orson Peters <orsonpeters@gmail.com>

pola-rs#27087) Co-authored-by: nameexhaustion <simonlin.rqmmw@slmail.me>

…s#27103)

… to "line" (pola-rs#27122)

…ola-rs#27102)

…a-rs#27123)

Co-authored-by: Dani Pinyol <dani@avatarcognition.com>

…la-rs#27118) Co-authored-by: Orson Peters <orsonpeters@gmail.com>

…7117)

…s#27144)

Resolve conflict in polars-stream/Cargo.toml: keep both hf_bucket_sink feature (ours) and is_first_distinct feature (upstream). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Replace the custom HfBucketSinkNode (1,050 lines of XET-specific code) with a standard ObjectStore implementation backed by OpenDAL's HF service. HF URLs now flow through the same FileSink path as S3/GCS/Azure, requiring only a thin build_hf() builder in a new hf.rs module. Key changes: - Add crates/polars-io/src/cloud/hf.rs: HF URL parsing, token extraction, and OpenDAL ObjectStore construction (~175 lines) - Wire CloudType::Hf in object_store_setup.rs to call build_hf(), matching the pattern used by build_aws/build_gcp/build_azure - Delete custom sink: hf_bucket/ directory (4 files, 721 lines), HfBucketSinkNode (260 lines), IR lowering special-case, PhysNodeKind::HfBucketSink variant - Rename feature flag hf_bucket_sink -> hf across 9 Cargo.toml files - Bump object_store_opendal compatibility from object_store 0.12 to 0.13 Dependencies: opendal + object_store_opendal (local path deps for now, will switch to published crate versions once apache/opendal#7185 ships). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

davanstrien · 2026-04-02T12:00:37Z

Replaced by a clean branch — see new PR.

Kevin-Patyk and others added 30 commits March 13, 2026 21:35

fix(python): Allow DataTypeExpr in pl.lit() (pola-rs#26740)

7bdf20d

fix: Fix panic on lazy concat->filter->slice with CSPE (pola-rs#26907)

8d54d5a

docs: Query Profiler addition to User Guide (pola-rs#26623)

84147c0

build(rust): Bump up numpy and pyo3 to 0.28 (pola-rs#26743)

a5893d7

fix: Propagate null in min_by / max_by for all-null by groups (po…

9d3ef6d

…la-rs#26919)

fix: Fix error passing Series of dates to business functions (pola-…

bfc0bc2

…rs#26927)

test(python): Set stricter maintain_order in `test_schema_row_index…

075e39a

…_cse` (pola-rs#26931)

fix: Default engine as streaming for collect_batches (pola-rs#26932)

f6b26bf

fix: Fix the loop bounds in `BitmapBuilder::extend_each_repeated_from…

a39cdb1

…_slice_unchecked` (pola-rs#26928)

fix(python): Fix search_sorted typing when used with list[float] (p…

f624da6

…ola-rs#26938) Co-authored-by: gabriel <gabriel.g.robin@airbus.com>

fix: Fix panic on upsample() with group_by parameter on empty DataFra…

9f23b42

…me (pola-rs#26936)

build: Bump lz4_flex from 0.12.0 to 0.12.1 (pola-rs#26940)

1c8174c

Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

fix: Fix ColumnNotFound due to projection between filter/cache in CSPE (

0f29993

pola-rs#26946)

fix: Follow-up on streaming range-join PR (pola-rs#26944)

bd50cb7

feat: Support casting Duration to String in ISO 8601 format (pola-rs#…

e70bda2

…26860)

perf: Optimize .replace() from a single value (pola-rs#26948)

150598d

feat: Support Decimal32/64 in scan_parquet (pola-rs#26941)

32b09f6

fix: Preserve height when unnesting empty struct columns (pola-rs#26947)

34a8d6e

Co-authored-by: moktamd <moktamd@users.noreply.github.com> Co-authored-by: nameexhaustion <simonlin.rqmmw@slmail.me>

feat: Support Delta deletion vectors in scan_delta (pola-rs#26867)

6be5f65

perf: Drop unused filter column above cache (pola-rs#26955)

bb50a78

perf: Native streaming forward_fill (pola-rs#26922)

642e185

feat: Error if PartitionBy path provider returns absolute path that d…

9b625c4

…oes not begin with base path, or contains '..' (pola-rs#26894)

feat: Truncate large binary/utf8 Parquet statistics values (pola-rs#2…

a83134d

…6764) Co-authored-by: Simon Lin <simonlin.rqmmw@slmail.me>

feat(python): Raise ChronoFormatWarning when using %f time format s…

a56f736

…tring (pola-rs#26934)

fix: Raise for duplicate columns in over() (pola-rs#26968)

7916b07

Co-authored-by: Claude <noreply@anthropic.com>

perf: Streaming native backward_fill (pola-rs#26967)

f967faa

perf: Lower index_of to streaming engine (pola-rs#26923)

e71947a

chore: Polars version (pola-rs#26971)

9a7f617

feat: Use UUIDv7 for sink_iceberg directory name generation (pola-rs#…

b7c51fc

…26958)

fix: Informative error for multi-quantile in group_by (pola-rs#26957)

8a5440b

Co-authored-by: nameexhaustion <simonlin.rqmmw@slmail.me>

MarcoGorelli and others added 22 commits March 30, 2026 14:48

chore(python): Improve internal typing ahead of using ty / pyrefly (

5802933

pola-rs#27050)

perf: Take into account size per row in join sampling (pola-rs#27098)

55badc4

fix: Preserve casts for horizontal ops with untyped literals (pola-rs…

9bc2600

…#27011)

docs: Put first-time contribution requirements in its own linkable se…

7b5f667

…ction (pola-rs#27113)

fix: Ignore ddof parameter in rolling_corr and deprecate (pola-rs…

5d8becc

…#27104) Co-authored-by: Orson Peters <orsonpeters@gmail.com>

fix: Apply scalar bound in clip when the Series bound contains nulls (

72fca32

pola-rs#27087) Co-authored-by: nameexhaustion <simonlin.rqmmw@slmail.me>

refactor(rust): Add memory usage tracking to global allocator (pola-r…

8d87d94

…s#27103)

feat(python): Change default scan/read_lines column name from "lines"…

5ecfa35

… to "line" (pola-rs#27122)

docs: Make the files used in docs available locally (pola-rs#27121)

7ce0ba6

perf: Use delta stats for mixed hive and non-hive predicate pushdown (p…

cd7361b

…ola-rs#27102)

fix: Fix incorrect IO metrics on multi-phase streaming execution (pol…

a0a98af

…a-rs#27123)

perf: Remove unused expression sorts (pola-rs#27075)

c2a0cec

chore(python): Add None & Dataframe to FrameInitTypes (pola-rs#27126)

eac36c5

Co-authored-by: Dani Pinyol <dani@avatarcognition.com>

fix(python): Address a potential overflow in from_epoch scaling (po…

5e06fbf

…la-rs#27118) Co-authored-by: Orson Peters <orsonpeters@gmail.com>

chore(python): Make internal typing more precise (part ii) (pola-rs#2…

d986cea

…7117)

refactor: Simplify pyarrow scan and process in batches (pola-rs#26982)

fefd58a

fix: Skip extension types for min/max in describe (pola-rs#27120)

c94f88b

fix: Output SVG if output_path ends with '.svg' in show_graph (pola-r…

45ee6d6

…s#27144)

chore: Update nightly Rust compiler version (pola-rs#27145)

b509848

feat: Allow group_by() without key exprs (pola-rs#27141)

5053689

Merge upstream/main into refactor/opendal-hf-sink

fdf3e21

Resolve conflict in polars-stream/Cargo.toml: keep both hf_bucket_sink feature (ours) and is_first_distinct feature (upstream). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

github-actions Bot added internal python rust A-dtype-object title needs formatting changes-dsl labels Apr 2, 2026

davanstrien closed this Apr 2, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

refactor: replace custom XET sink with OpenDAL ObjectStore#2

refactor: replace custom XET sink with OpenDAL ObjectStore#2
davanstrien wants to merge 95 commits intofeature/hf-bucket-sinkfrom
refactor/opendal-hf-sink

davanstrien commented Apr 2, 2026

Uh oh!

davanstrien commented Apr 2, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants

Conversation

davanstrien commented Apr 2, 2026

Summary

Motivation

What's deleted

What's added

Dependencies

Test plan

Uh oh!

davanstrien commented Apr 2, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants