Skip to content

Fix examples by separating Mock vs Live workflows and adding CI validation #96

@blindzero

Description

@blindzero

Problem Statement

Examples are a critical release surface (v0.8.0 must not ship with broken examples).
Currently, examples/Invoke-IdleDemo.ps1 enumerates all workflows from examples/workflows/*.psd1, including workflows that require real systems (AD / Entra ID / Entra Connect). This causes:

  • default demo runs to select a non-mockable workflow (sorted first) and fail immediately
  • -List, -All, and interactive selection to include workflows that cannot run with IdLE.Provider.Mock
  • examples to appear "broken" even though unit tests are green
  • drift after breaking changes (e.g., step metadata/capabilities changes) to go unnoticed for example workflows

Proposed Solution

1) Explicit workflow categories (Mock vs Live vs Templates)

Introduce clear categories to remove ambiguity and keep demos deterministic:

  • Mock workflows: runnable out-of-the-box with IdLE.Provider.Mock (release gate)
  • Live workflows: example workflows intended to run with real providers/systems (not executed in CI)
  • Templates: starting points; must remain structurally valid but are not executed in CI

Recommended directory structure:

  • examples/workflows/mock/
  • examples/workflows/live/
  • examples/workflows/templates/

2) Demo script behavior (examples/Invoke-IdleDemo.ps1)

The demo entrypoint must only list/run mock workflows by default:

  • default behavior (no params): -Category Mock
  • -List default behavior: list only Mock
  • interactive selection default: only Mock

Support explicit category selection:

  • -Category Mock|Live|Templates|All

Notes:

  • Categories are an examples-level concern. Engine/Core does not "auto-discover" workflows.
  • Optional: keep Demo as an alias to Mock for backwards compatibility.

3) Workflow classification (to remove ambiguity)

Move existing workflows according to these rules:

Mock workflows (examples/workflows/mock/)

Must run with:

  • IdLE + IdLE.Steps.Common
  • IdLE.Provider.Mock using provider alias Identity

Candidate list (based on current filenames; adjust if needed):

  • joiner-minimal.psd1
  • joiner-minimal-ensureattribute.psd1
  • joiner-ensureentitlement.psd1
  • joiner-with-condition.psd1
  • joiner-with-onfailure.psd1

Live workflows (examples/workflows/live/)

Example workflows that require real providers/systems (AD/Entra/DirectorySync/Auth):

  • joiner-with-entraid-sync.psd1
  • ad-joiner-complete.psd1
  • ad-mover-department-change.psd1
  • ad-leaver-offboarding.psd1
  • entraid-joiner-complete.psd1
  • entraid-mover-department-change.psd1
  • entraid-leaver-offboarding.psd1

Templates (examples/workflows/templates/)

If workflows are intended as generic starting points and not as demonstrations, place them here.
(If no additional templates exist yet, this folder can still be created for future use.)

4) Documentation updates (in scope)

Update examples/README.md to clearly explain:

  • categories (Mock / Live / Templates)
  • how to run the demo script per category
  • prerequisites for Live workflows (documented, but not validated automatically in this issue)

Add a workflow matrix/table including at minimum:

  • workflow file
  • category (Mock/Live/Template)
  • runnable with Mock? (yes/no)
  • required provider aliases
  • external prerequisites / notes (for Live)

Update root README.md and relevant docs/quickstart wording to avoid implying that all workflows are runnable out-of-the-box.

5) CI validation and smoke tests (in scope)

Mock workflows: smoke tests (must pass; release gate)

Add Pester tests that enumerate examples/workflows/mock/*.psd1 and verify:

  • Test-IdleWorkflow passes
  • New-IdlePlan succeeds using IdLE.Provider.Mock
  • Invoke-IdlePlan completes successfully using IdLE.Provider.Mock

Live workflows: validation tests (must pass; not executed)

Add Pester tests that enumerate examples/workflows/live/*.psd1 and verify:

  • Test-IdleWorkflow passes

No connectivity/prerequisite validation and no execution in this issue.

Templates: validation tests (must pass; not executed)

Add Pester tests that enumerate examples/workflows/templates/*.psd1 and verify:

  • Test-IdleWorkflow passes

No execution in this issue.

6) Align workflows with recent breaking changes (in scope)

Ensure all example workflows are compatible with the current engine contracts, including recent changes such as:

  • step metadata / capabilities ownership (workflows must not specify RequiredCapabilities / RequiresCapabilities)
  • step pack catalogs and step type naming consistency
  • provider alias usage and other schema validations

Alternatives Considered

  1. Naming convention filter (e.g., mock-*.psd1)
  • Rejected: less explicit and easier to accidentally regress.
  1. Keep listing all workflows in the demo, but handle failures
  • Rejected: poor onboarding experience and non-deterministic behavior.
  1. Auto-detect mock-runnable workflows by planning each workflow
  • Rejected: slow/noisy and introduces implicit behavior into the demo script.

Impact

  • Examples folder structure changes (paths change), limited to examples/docs/tests.
  • Better onboarding: default demo is deterministic and runnable without lab.
  • Release quality is measurable via CI for Mock workflows.
  • Live and Template workflows are kept valid via CI validation.

Acceptance Criteria

  • examples/Invoke-IdleDemo.ps1 lists/runs Mock workflows by default.
  • -Category Mock|Live|Templates|All is supported (and consistent for -List, -All, interactive selection).
  • Workflows are separated into mock/, live/, and templates/.
  • Mock workflows pass CI smoke tests (validate + plan + execute) with IdLE.Provider.Mock.
  • Live workflows pass CI validation (Test-IdleWorkflow).
  • Template workflows pass CI validation (Test-IdleWorkflow).
  • Examples/docs are updated and clearly explain categories, prerequisites, and how to run.
  • All example workflows are updated to match current engine contracts (e.g. no workflow-defined capabilities).
  • Pester green; ScriptAnalyzer green.

Additional Context / Implementation Notes

  • Keep category handling in the examples layer (demo script + folder structure); do not add host/framework coupling to IdLE.Core.
  • For Live prerequisites (AD/Entra/DirectorySync/Auth), document expected setup in examples/README.md. Validation/execution for Live workflows can be addressed in a later issue (e.g., integration lab / self-hosted runner).

Metadata

Metadata

Labels

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions