Add replicas parameter to PipelineStep macro#231
Conversation
Adds a new `replicas` parameter to the PipelineStep macro to control the number of replicas in generated Kubernetes deployments. The parameter defaults to 1 to maintain backward compatibility with existing usage. - Add replicas field to PipelineStepContext TypedDict - Update parse_context to handle replicas with default value of 1 - Set replicas in deployment spec during manifest generation - Add comprehensive test coverage for default and custom values - Update docstring example to demonstrate usage All tests pass and type checking succeeds.
Semver Impact of This PR⚪ None (no version bump detected) 📋 Changelog PreviewThis is how your changes will appear in the changelog. New Features ✨
Other
🤖 This preview updates automatically when you update the PR. |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
Refactored conflict detection to use fail_on_scalar_overwrite parameter in deepmerge instead of separate validation logic. This approach: - Automatically detects all scalar field conflicts without maintaining explicit lists - Provides accurate path tracking for better error messages - Detects conflicts with both user template and base template values - Allows dict merging and list appending as expected - Only fails when scalar values actually differ Changes: - Add fail_on_scalar_overwrite parameter to deepmerge function - Add ScalarOverwriteError exception with path tracking - Remove TemplateConflictError and detect_scalar_overwrites helper - Add 7 comprehensive tests for new behavior in test_merge.py - Consolidate to single conflict test in test_pipeline_step.py All tests pass (41 total). Co-authored-by: Cursor <cursoragent@cursor.com>
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 2 potential issues.
Bugbot Autofix is OFF. To automatically fix reported issues with Cloud Agents, enable Autofix in the Cursor dashboard.
| f"Cannot merge key '{key}': base type is {type(base_value)} but override type is {type(override_value)}" | ||
| ) | ||
| else: | ||
| # Scalar to scalar replacement |
There was a problem hiding this comment.
TypeMismatchError lacks full path in error message
Low Severity
The _path tracking and path_str computation were added in this PR for better error messages, but TypeMismatchError still uses only the immediate key while ScalarOverwriteError uses the full path_str. This results in inconsistent error message quality—a deeply nested type mismatch shows only "Cannot merge key 'containers'" while a scalar overwrite shows "Cannot overwrite scalar at 'spec.template.spec.containers'". The path_str is also computed on every iteration but unused for most cases.
2 participants
Summary
This PR adds a
replicasparameter to thePipelineStepmacro insentry_streams_k8sto support parallelism in Kafka streaming applications.Ideally we should support something like this https://getsentry.github.io/streams/configure_pipeline.html. But that requires some design discussion about the semantics of the replicas field.
In the meantime we just need a way to set the replicas count on the k8s deployment. Adding the replicas count in the macro rather than in the config makes it easy to move it later.
Usage Example
{{ render_external( "sentry_streams_k8s.pipeline_step.PipelineStep", { "service_name": "my-service", "pipeline_name": "profiles", "replicas": 3, # NEW: Set number of replicas # ... other parameters } ) }}