Skip to content

Add working pipeline system implementation #27

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
brian-arnold merged 56 commits into from
Jul 3, 2025
Merged

Add working pipeline system implementation #27

brian-arnold merged 56 commits into from
Jul 3, 2025

Conversation

@eywalker
Copy link
Contributor

@eywalker eywalker commented Jun 26, 2025

  • Full updated pipeline framework is now implemented

@eywalker eywalker requested a review from brian-arnold June 26, 2025 18:09
@eywalker eywalker marked this pull request as draft June 27, 2025 17:43
eywalker added 23 commits July 1, 2025 06:53
@eywalker
refactor: cleanup imports and use versioned object hasher
ba1f45d
@eywalker
fix: failure to reset cache due to mro mixup
e689d0d
@eywalker
style: apply ruff format
6222064
@eywalker
fix: legacy_core imports
cbe82ab
@eywalker
wip: arrow logical serialization
caca67b
@eywalker
refactor: utils renaming and relocation
7bc98e1
@eywalker
fix: cleanup imports and fix issue in recursive structure processing
51f3da2
@eywalker
refactor: add more robust arrow serialization strategy and use @ for …
3d54067 
…separator in hash
@eywalker
feat: logical serialization for arrow table
1ac2be6
@eywalker
feat: update versioned arrow hasher to use new serialization
dab3378
@eywalker
wip: delta table store implementation
4f07927
@eywalker
feat: better handling of stores and add flushing to stores and pipeline
1b7519e
@eywalker
feat: integrate actual saving to parquet into simple in memory store
07fd76e
@eywalker
refactor: cleanup improt and comment out old packet converter for fut…
8411b40 
…ure removal
@eywalker
fix: attach label on kernel invocation to the invocation object
d90e5c6
@eywalker
fix: invoke superclass init
fe35aba
@eywalker
feat: expose explicit check for assigned label on content identifiabl…
ef301b3 
…e base
@eywalker
feat: add label on wrapped invocation
ead6704
@eywalker
doc: add tutorial notebook
cbb8754
@eywalker
refactor: clean up store package
73b2638
@eywalker
feat: improve pipeline usability with typechecks and convenience attr…
555a751 
…ibutes
@eywalker
fix: use new store name
083134b
@eywalker
test: update to use new package name
7e33bae
@review-notebook-app
Copy link

review-notebook-app bot commented Jul 3, 2025

Check out this pull request on  ReviewNB

See visual diffs & provide feedback on Jupyter Notebooks.

Powered by ReviewNB

@eywalker eywalker changed the base branch from main to pipeline July 3, 2025 21:46
@eywalker eywalker marked this pull request as ready for review July 3, 2025 21:47
@eywalker eywalker requested a review from brian-arnold July 3, 2025 21:47
@brian-arnold brian-arnold merged commit 00b4066 into walkerlab:pipeline Jul 3, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@brian-arnold brian-arnold brian-arnold approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@eywalker @brian-arnold