Step groups #53
Description
Is your feature request related to a problem? Please describe.
Sometimes multiple steps are needed to represent a single logical transformation in a pipeline. It would be nice if these could all be built and imported into the pipeline as one logical unit.
Describe the solution you'd like
I am proposing creating a Sequence object that holds one or more steps. A Sequence can be imported into a pipeline the same way steps are. To avoid making breaking changes to how the pipeline works, this will only affect pipeline construction, as once the pipeline starts these are just normal steps.
Describe alternatives you've considered
Another alternative would be to have the Sequence class inherit from the step class and run all of its steps internally. I think this is a poor design choice for the following reasons:
- Metrics would break as per-step measurements are no longer atomic.
- Step failures would be harder to recover from.
- Locks are handled at the pipeline level per step, meaning that this increases the risk of expired locks due to extended runtime.
Additional context
N/A