Skip to content

Add parallel step execution in YAML workflows #4

@tombee

Description

@tombee

Problem

YAML workflows currently execute steps sequentially. Many workflows have independent steps that could run concurrently, reducing total execution time.

Concrete example from foreman: The gathering stage could run codebase analysis and documentation reading in parallel, cutting stage time in half.

Proposal

Add parallel step execution support:

steps:
  - id: parallel_analysis
    parallel:
      - id: analyze_code
        type: llm
        prompt: "Analyze the codebase structure"
      
      - id: read_docs
        type: llm  
        prompt: "Summarize the documentation"
    
  - id: combine
    type: llm
    prompt: |
      Combine findings:
      Code: {{.steps.analyze_code.response}}
      Docs: {{.steps.read_docs.response}}

General Utility

  1. Multi-source analysis - Gather information from multiple sources simultaneously
  2. Batch processing - Process multiple items in parallel
  3. Independent validations - Run checks concurrently

Acceptance Criteria

  • parallel step type that runs child steps concurrently
  • All parallel steps complete before workflow continues
  • Errors in any parallel step fail the parallel block (configurable)
  • Step outputs accessible via normal template syntax
  • Token tracking aggregates across parallel steps
  • Integration test for parallel execution

Priority

Medium - Performance improvement, not blocking for MVP.

Labels

enhancement, sdk, priority:medium

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions