Engine runtime validation: reuse wfctl cross-reference checks at startup and execution time

[intel352]

Summary

Follow-up to #368 (wfctl static cross-reference validation) and #374 (output type enforcement).

#368 proposes deep template reference validation in wfctl validate — checking field-level references against step output schemas, SQL alias extraction, and cross-file import validation. That same validation logic should also be available inside the engine itself, so that problems are caught automatically at startup or first execution without requiring a separate wfctl validate CI step.

Motivation

• Not every team runs wfctl validate in CI — the engine should protect itself
• Static analysis catches config-time issues, but some references are only resolvable at runtime (dynamic SQL, template-generated step names, per-tenant configs)
• A config that passes wfctl validate could still fail at runtime due to environment-specific differences (env var expansion, imported file resolution)
• Developers iterating locally benefit from immediate feedback on startup rather than waiting for the first request to hit a broken pipeline

Proposed behavior

Phase 1: Startup validation (config load time)

When the engine loads config via BuildFromConfig(), run the same checks that wfctl template validate performs:

1. Step name references — already done in wfctl, reuse in engine
2. Step output field references — validate that steps.query.row references a field declared in step.db_query's output schema (proposed in wfctl validate: no cross-reference checking for DB columns, step output fields, or imported configs #368)
3. Cross-file imports — verify all referenced pipelines/steps exist in the merged config
4. Mode-dependent output validation — e.g., steps.query.rows on a mode: single db_query

This should be warnings by default to maintain backwards compatibility:

engine:
  validation:
    template_refs: warn   # "off" | "warn" | "error" (default: "warn")

With warn (default): log warnings at startup for any suspicious references, but allow the engine to start. With error: refuse to start if validation fails. With off: skip entirely (current behavior).

Phase 2: First-execution validation (runtime)

Some references can only be validated when a pipeline actually runs — for example, when step outputs are known. On first execution of each pipeline, the engine could:

1. After each step completes, validate that its actual output keys match the declared StepSchema.Outputs
2. Before resolving a template, check that the referenced step has already produced the expected output keys
3. Log warnings (or error, based on config) when a template resolves a missing key to zero value

This provides a runtime safety net that catches issues static analysis cannot:

WARN [pipeline=get-form step=respond] template references steps.query.row.slug
     but step "query" output keys are: [row, found] — "slug" is a sub-key of "row"
     which cannot be statically verified (depends on SQL result)

Implementation: share validation logic

The key principle is one validation library, two consumers:

wfctl/template_validate.go  ──┐
                               ├── shared validation package
engine/pipeline_executor.go ──┘

The validation logic proposed in #368 (deep template ref checking, output schema lookup, SQL alias extraction) should live in a shared package (e.g., validation/ or within config/) that both wfctl and the engine import. This avoids duplicating logic and ensures consistency.

// validation/template_refs.go (shared)
type RefValidationResult struct {
    Warnings []string
    Errors   []string
}

func ValidateTemplateRefs(pipelines map[string]PipelineConfig, schemas map[string]StepSchema) *RefValidationResult

// Used by wfctl:
//   result := validation.ValidateTemplateRefs(cfg.Pipelines, registry.Schemas())
//   printWarnings(result)

// Used by engine at startup:
//   result := validation.ValidateTemplateRefs(cfg.Pipelines, registry.Schemas())
//   if engineConfig.Validation.TemplateRefs == "error" && len(result.Errors) > 0 {
//       return fmt.Errorf("template validation failed: %v", result.Errors)
//   }
//   for _, w := range result.Warnings { logger.Warn(w) }

Backwards compatibility

• Default behavior (warn) only adds log output — no functional change
• off preserves exact current behavior
• error is opt-in for teams that want strict validation
• No existing configs break

References

• wfctl validate: no cross-reference checking for DB columns, step output fields, or imported configs #368 — wfctl static cross-reference validation (the logic to reuse)
• Pipeline and step output type schemas are metadata-only — no runtime or static enforcement #374 — output type schema enforcement (declares the schemas to validate against)
• Template missingkey=zero silently swallows field-level typos in step references #367 — missingkey=zero (the runtime symptom this would surface as warnings)
• Engine config loading: engine.go / BuildFromConfig()
• Template validation logic: cmd/wfctl/template_validate.go
• Step output schemas: schema/step_schema_builtins.go (182 step types)