Open
Conversation
Contributor
Test Coverage Report
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
…assflow-python-sdk into add-support-for-pipeline-v3
Contributor
Test Coverage Report
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Restructure the SDK models to match the new V3 pipeline specification:
- sources: flat list of independent source objects (each with own
connection_params and single topic) replaces nested source.topics
- transforms: unified list of {type, source_id, config} entries
replaces separate filter, stateless_transformation, and per-topic
deduplication fields
- join: explicit left_source/right_source with output_fields replaces
sources list with orientation
- resources: renamed from pipeline_resources, with sources list
(replaces ingestor) and per-source_id transform list
Also fixes: NullPointerError in resource update methods, missing
source_id uniqueness validation, duplicate data_types mapping key,
and missing @classmethod on Pydantic validator.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Contributor
Test Coverage Report
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Contributor
Test Coverage Report
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
The backend v3.0.0 release ships a POST /api/v1/pipeline/migrate-preview
endpoint that converts v2 pipeline configs to v3 server-side. The SDK
no longer needs its own local conversion logic.
Changes:
- Add Client.migrate_pipeline_v2_to_v3(config) that POSTs to the new
endpoint and returns a validated PipelineConfig. Same signature as
the removed helper so callers only update the import site.
- Remove src/glassflow/etl/utils.py (the old local helper) and
tests/test_utils.py (its tests).
- Drop migrate_pipeline_v2_to_v3 from the etl package __all__ export.
- Update the v2-rejection error in PipelineConfig.validate_version to
suggest the new Client method:
Pipeline version v2 is no longer supported by this SDK. Convert
your v2 configuration to v3 by calling
`glassflow.etl.Client().migrate_pipeline_v2_to_v3(config)`,
then pass the returned config to create_pipeline(). Alternatively,
pin glassflow-python-sdk<4.0.0 to keep working with v2 pipelines
directly.
The v1 rejection is now a separate message (no Client hint — no
migration path for v1).
- Add two unit tests for the new Client method: success path asserts a
POST to /api/v1/pipeline/migrate-preview with the v2 body and a
validated PipelineConfig return; error path asserts APIError
surfaces.
Verified on the staging cluster (http://localhost:8081 port-forward):
the endpoint correctly migrates a real v2 Kafka+dedup pipeline to v3
(sources[], transforms[], nested sink.connection_params, sink.mapping
from top-level schema). The full SDK smoke test across Client,
Pipeline, and DLQ methods passes (17/17) including create from dict
and YAML file, get, rename, to/from YAML and JSON, stop, resume,
terminate, delete, and DLQ state/consume.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Collapsed the migrate test signature onto a single line per ruff format; no behavior change. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Contributor
Test Coverage Report
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Contributor
Test Coverage Report
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.