Add working pipeline system implementation #27

Merged

brian-arnold merged 56 commits intowalkerlab:pipelinefrom

eywalker:pipeline

Jul 3, 2025

Commits on Jun 19, 2025

Commits on Jun 24, 2025

Commits on Jun 26, 2025

Commits on Jun 27, 2025

Commits on Jun 30, 2025

wip: use new schema system
eywalker
committed

Commits on Jul 1, 2025

Commits on Jul 2, 2025

fix: cleanup imports and fix issue in recursive structure processing
eywalker
committed
refactor: add more robust arrow serialization strategy and use @ for separator in hash
eywalker
committed
feat: logical serialization for arrow table
eywalker
committed
feat: update versioned arrow hasher to use new serialization
eywalker
committed
wip: delta table store implementation
eywalker
committed
feat: better handling of stores and add flushing to stores and pipeline
eywalker
committed
feat: integrate actual saving to parquet into simple in memory store
eywalker
committed
refactor: cleanup improt and comment out old packet converter for future removal
eywalker
committed

Commits on Jul 3, 2025