feat: background route dispatch with join synchronization

[PolyphonyRequiem]

Provenance: AI-drafted (GitHub Copilot CLI / Claude Opus 4.6), human-reviewed and approved, AI-submitted.

Problem

Conductor workflows execute agents strictly sequentially — each agent must complete before the next begins. This is correct for most patterns, but creates unnecessary bottlenecks when independent work could overlap.

Concrete example: In an SDLC workflow, after implementing all tasks in a PR group, the workflow runs a PR lifecycle (reduce → submit → review → merge) that takes 30 minutes. During this time, the next PR group's tasks could start on a different branch. Today they wait.

Existing primitives don't solve this:

• parallel groups are static (all agents defined at YAML time, execute concurrently, no routing)
• for_each groups are homogeneous (same agent, many items)
• Neither supports "start this agent, don't wait, continue with other routing"

Proposed solution: mode: background on routes

A route annotation that starts the target agent/sub-workflow asynchronously:

- name: pr_group_manager
  routes:
    - to: pr_lifecycle
      when: "{{ output.action == 'submit_pr' }}"
      mode: background          # start but don't block
    - to: task_manager
      when: "{{ output.action == 'start_tasks' }}"   # runs immediately
    - to: join_prs
      when: "{{ output.action == 'all_complete' }}"

A new join agent type that waits for all background instances to complete:

- name: join_prs
  type: join
  wait_for: [pr_lifecycle]
  failure_mode: continue_on_error   # fail_fast | continue_on_error | all_or_nothing
  output:
    completed: { type: array }
    errors: { type: array }
    total: { type: number }
  routes:
    - to: pr_finalizer

Semantics

Background dispatch:

• When a route has mode: background, the engine starts the target agent as an asyncio.Task with a frozen context snapshot
• The main loop immediately evaluates the next matching route (first non-background match, or fall-through)
• Multiple background routes can fire from the same agent execution
• Background agents store their outputs in a background-specific context namespace

Join collection:

• wait_for lists agent names that may have been dispatched as background
• The join blocks until all instances of those agents complete
• failure_mode controls error handling (same semantics as for_each):
  • continue_on_error: collect all results, fail only if ALL background tasks failed
  • fail_fast: cancel remaining background tasks on first failure
  • all_or_nothing: wait for all, fail if any failed
• Join output provides completed (successful outputs), errors (failures), and total count

Context isolation:

• Each background agent gets a deep copy of the context at dispatch time
• Background agents cannot see each other's outputs or the main loop's subsequent state
• The join output aggregates results for downstream agents

Why these semantics

• Context snapshots prevent race conditions — no shared mutable state
• continue_on_error default because real workflows have irreversible operations (merged PRs can't be un-merged)
• Join as explicit agent keeps the workflow graph readable — you can see where convergence happens
• Multiple instances are tracked by agent name + dispatch index, similar to for_each item tracking

Design decisions

Decision	Choice	Why
Annotation on routes vs new agent type	Route annotation	Background is a routing concern, not an agent concern — same agent can be foreground or background depending on context
Context sharing	Snapshot at dispatch	Mutable sharing would require locking; snapshots are simpler and match sub-workflow semantics
Join syntax	Explicit agent	Implicit joins (auto-wait at $end) would make control flow invisible
Failure handling	Same as for_each	Proven pattern, users already understand it

Non-goals (v1)

• Background agents seeing each other's outputs (they're isolated)
• Dynamic join (waiting for a computed list of agents)
• Background sub-workflows in for_each (combine later if needed)
• Cancellation of background tasks from the main loop (except via fail_fast on join)

Estimated scope

Schema: Add mode to RouteDef, add JoinDef as new agent type
Engine: Modify execution loop to spawn asyncio tasks for background routes, add join collection logic
Validator: Validate join references, warn on background routes without a downstream join
Events: New event types: background_started, background_completed, background_failed, join_waiting, join_completed
Tests: Background dispatch, join collection, failure modes, context isolation