Document YAML-first templating pipeline

Makefile

@@ -12,7 +12,7 @@ release: target/release/$(APP) ## Build release binary
 
 all: release ## Default target builds release binary
 
-clean: ## Remove build artifacts
+clean: ## Remove build artefacts
 	$(CARGO) clean
 
 test: ## Run tests with warnings treated as errors

docs/behavioural-testing-in-rust-with-cucumber.md

@@ -979,7 +979,7 @@ The process involves two main steps:
 2. **Publish reports:** Many CI platforms can parse and display test results
    in a structured format. The `cucumber` crate supports generating JUnit XML
    reports via the `output-junit` feature flag.[^16] These XML files can then
-   be published as test artifacts for platforms like GitHub Actions, GitLab
+   be published as test artefacts for platforms like GitHub Actions, GitLab
    CI,[^34] or Jenkins to consume.[^33]
 
 This CI integration closes the BDD loop. The `.feature` files, once checked

docs/netsuke-design.md

@@ -31,35 +31,42 @@ understand and execute. This separation of concerns—Netsuke managing build
 logic and Ninja managing execution—is the foundational principle of the entire
 architecture.
 
-### 1.2 The Five Stages of a Netsuke Build
+### 1.2 The Six Stages of a Netsuke Build
 
 The process of transforming a user's `Netsukefile` manifest into a completed
-build artifact follows a distinct, five-stage pipeline. This multi-stage data
-flow ensures that dynamic configurations are fully resolved into a static plan
+build artefact now follows a six-stage pipeline. This data flow validates the
+manifest as YAML first, then resolves all dynamic logic into a static plan
 before execution, a critical requirement for compatibility with Ninja.
 
 1. Stage 1: Manifest Ingestion
 
    The process begins by locating and reading the user's project manifest file
    (e.g., Netsukefile) from the filesystem into memory as a raw string.
 
-2. Stage 2: Jinja Evaluation
+2. Stage 2: Initial YAML Parsing
 
-   The raw manifest string is treated as a Jinja template. Netsuke's templating
-   engine processes this string, evaluating all expressions, executing control
-   structures (loops, conditionals), and applying filters. This stage resolves
-   all dynamic aspects of the build, producing a static, pure YAML string as
-   its output.
+   The raw string is parsed into an untyped `serde_yml::Value`. This step
+   ensures the manifest is valid YAML before any templating takes place.
 
-3. Stage 3: YAML Parsing & Deserialization
+3. Stage 3: Template Expansion
 
-   The static YAML string generated in the previous stage is passed to a YAML
-   parser. This parser validates the YAML syntax and deserializes the content
-   into a set of strongly typed Rust data structures. This collection of
-   structs, which directly mirrors the YAML schema, can be considered an
-   "unprocessed" Abstract Syntax Tree (AST) of the build plan.
+   Netsuke walks the YAML `Value`, evaluating Jinja macros, variables, and the
+   `foreach` and `when` keys. Each mapping containing these keys is expanded
+   with an iteration context providing `item` and optional `index`. Variable
+   lookups respect the precedence `globals` < `target.vars` < per-iteration
+   locals, and this context is preserved for later rendering. At this stage
+   Jinja must not modify the YAML structure directly; control constructs live
+   only within these explicit keys. Structural Jinja blocks (`{% ... %}`) are
+   not permitted to reshape mappings or sequences.
 
-4. Stage 4: IR Generation & Validation
+4. Stage 4: Deserialisation & Final Rendering
+
+   The expanded `Value` is deserialised into strongly typed Rust structs. Jinja
+   expressions are then rendered, but only within string fields. Structural
+   templating using `{% %}` blocks is forbidden; all control flow must appear
+   in YAML values.
+
+5. Stage 5: IR Generation & Validation
 
    The AST is traversed to construct a canonical, fully resolved Intermediate
    Representation (IR) of the build. This IR represents the build as a static
@@ -70,7 +77,7 @@ before execution, a critical requirement for compatibility with Ninja.
    exactly one of `rule`, `command`, or `script`. Circular dependencies and
    missing inputs are also detected at this stage.
 
-5. Stage 5: Ninja Synthesis & Execution
+6. Stage 6: Ninja Synthesis & Execution
 
    The final, validated IR is traversed by a code generator. This generator
    synthesizes the content of a `build.ninja` file, translating the IR's nodes
@@ -84,18 +91,12 @@ before execution, a critical requirement for compatibility with Ninja.
    output suitable for caching or source control.
 
 ```mermaid
-sequenceDiagram
-    participant User
-    participant Netsuke
-    participant IR_Generator
-    participant Ninja
-
-    User->>Netsuke: Provide Netsukefile and environment
-    Netsuke->>IR_Generator: Parse and validate manifest
-    IR_Generator->>IR_Generator: Generate deterministic BuildGraph (IR)
-    IR_Generator->>Netsuke: Return BuildGraph (byte-for-byte deterministic)
-    Netsuke->>Ninja: Synthesize build.ninja and invoke Ninja
-    Ninja-->>User: Execute build and report results
+flowchart TD
+    A[Stage 1:\nManifest Ingestion] --> B[Stage 2:\nInitial YAML Parsing]
+    B --> C[Stage 3:\nTemplate Expansion]
+    C --> D[Stage 4:\nDeserialisation & Final Rendering]
+    D --> E[Stage 5:\nIR Generation & Validation]
+    E --> F[Stage 6:\nNinja Synthesis & Execution]
 ```
 
 ### 1.3 The Static Graph Mandate
@@ -120,8 +121,8 @@ Representation (IR) generated in Stage 4. The IR serves as a static snapshot of
 the build plan after all Jinja logic has been resolved. It is the "object code"
 that the Netsuke "compiler" produces, which can then be handed off to the Ninja
 "assembler" for execution. This mandate for a pre-computed static graph
-dictates the entire five-stage pipeline and establishes a clean boundary
-between the user-facing logic layer and the machine-facing execution layer.
+dictates the entire six-stage pipeline and establishes a clean boundary between
+the user-facing logic layer and the machine-facing execution layer.
 
 ## Section 2: The Netsuke Manifest: A User-Centric YAML Schema
 
@@ -336,19 +337,28 @@ are specified.
 
 Large sets of similar outputs can clutter a manifest when written individually.
 Netsuke supports a `foreach` entry within `targets` to generate multiple
-outputs succinctly. The expression assigned to `foreach` is evaluated during
-the Jinja render phase, and each value becomes `item` in the target context.
+outputs succinctly. The `foreach` and optional `when` keys accept bare Jinja
+expressions evaluated after the initial YAML pass. Each resulting value becomes
+`item` in the target context, and the per-iteration environment is carried
+forward to later rendering.
 
 ```yaml
-- foreach: "{{ glob('assets/svg/*.svg') }}"
+- foreach: glob('assets/svg/*.svg')
+  when: item | basename != 'logo.svg'
   name: "{{ outdir }}/{{ item | basename | replace('.svg', '.png') }}"
   rule: rasterise
   sources: "{{ item }}"
 ```
 
-Each element in the sequence produces a separate target. The resulting build
-graph is still fully static and behaves the same as if every target were
-declared explicitly.
+Each element in the sequence produces a separate target. The iteration context:
+
+- `item`: current element
+- `index`: 0-based index (optional)
+- Variables resolve with precedence `globals` < `target.vars` < iteration locals
+
+Jinja control structures cannot shape the YAML; all templating must occur
+within the string values. The resulting build graph is still fully static and
+behaves the same as if every target were declared explicitly.
 
 ### 2.6 Table: Netsuke Manifest vs. Makefile
 
@@ -1426,13 +1436,13 @@ treated as the default subcommand if none is provided, allowing for the common*
 The behaviour of each subcommand is clearly defined:
 
 - `Netsuke build [--emit FILE] [targets...]`: This is the primary and default
-  command. It executes the full five-stage pipeline: ingestion, Jinja
-  rendering, YAML parsing, IR generation, and Ninja synthesis. By default the
-  generated Ninja file is written to a securely created temporary location and
-  removed after the build completes. Supplying `--emit FILE` writes the Ninja
-  file to `FILE` and retains it. If no targets are provided on the command
-  line, the targets listed in the `defaults` section of the manifest are
-  built.
+command. It executes the full six-stage pipeline: Manifest Ingestion, Initial
+YAML Parsing, Template Expansion, Deserialisation & Final Rendering, IR
+Generation & Validation, and Ninja Synthesis & Execution. By default the
+generated Ninja file is written to a securely created temporary location and
+removed after the build completes. Supplying `--emit FILE` writes the Ninja
+file to `FILE` and retains it. If no targets are provided on the command line,
+the targets listed in the `defaults` section of the manifest are built.
 
 - `Netsuke clean`: This command provides a convenient way to clean the build
   directory. It will invoke the Ninja backend with the appropriate flags, such
@@ -1495,7 +1505,7 @@ goal.
 
   - **Success Criterion:** Netsuke can successfully take a `Netsukefile` file
     *without any Jinja syntax* and compile it to a `build.ninja` file, then
-    execute it to produce the correct artifacts. This phase validates the
+    execute it to produce the correct artefacts. This phase validates the
     entire static compilation pipeline.
 
 - **Phase 2: The Dynamic Engine**
@@ -1565,7 +1575,7 @@ powerful build tool. The use of a decoupled IR, in particular, opens up many
 possibilities for future enhancements beyond the initial scope.
 
 - **Advanced Caching:** While Ninja provides excellent file-based incremental
-  build caching, Netsuke could implement a higher-level artifact caching layer.
+  build caching, Netsuke could implement a higher-level artefact caching layer.
   This could involve caching build outputs in a shared network location (e.g.,
   S3) or a local content-addressed store, allowing for cache hits across
   different machines or clean checkouts.

docs/roadmap.md

@@ -79,19 +79,24 @@ configurations with variables, control flow, and custom functions.
 
   - [x] Integrate the `minijinja` crate into the build pipeline.
 
-  - [x] Implement the two-pass parsing mechanism: the first pass renders the
-    manifest as a Jinja template, and the second pass parses the resulting pure
-    YAML string with serde_yml.
+  - [x] Implement data-first parsing: parse the manifest into a
+    `serde_yml::Value` (Stage 2: Initial YAML Parsing), expand `foreach` and
+    `when` entries with a Jinja environment (Stage 3: Template Expansion), then
+    deserialise the expanded tree into the typed AST and render remaining
+    string fields (Stage 4: Deserialisation & Final Rendering).
 
   - [x] Create a minijinja::Environment and populate its initial context with
     the global vars defined in the manifest.
 
 - [ ] **Dynamic Features and Custom Functions:**
 
-  - [x] Implement support for basic Jinja control structures (`{% if %}` and
-        `{% for %}`)
+  - [x] Evaluate Jinja expressions only within string values, forbidding
+        structural tags such as `{% if %}` and `{% for %}`.
 
-  - [ ] Implement the foreach key for target generation.
+  - [ ] Implement the `foreach` and `when` keys for target generation,
+        exposing `item` and optional `index` variables and layering
+        per-iteration locals over `target.vars` and manifest globals for
+        subsequent rendering phases.
 
   - [ ] Implement the essential custom Jinja function env(var_name) to read
     system environment variables.
@@ -105,8 +110,9 @@ configurations with variables, control flow, and custom functions.
 - **Success Criterion:**
 
   - [ ] Netsuke can successfully build a manifest that uses variables,
-    conditional logic, the foreach loop, custom macros, and the glob() function
-    to discover and operate on source files.
+    conditional logic within string values, the `foreach` and `when` keys,
+    custom macros, and the `glob()` function to discover and operate on source
+    files.
 
 ## Phase 3: The "Friendly" Polish 🛡️
 

scripts/assert-file-absent.sh

@@ -6,7 +6,7 @@ set -euo pipefail
 
 if [[ $# -ne 1 ]]; then
   echo "Usage: $(basename "$0") <file>" >&2
-  exit 2   # usage error
+  exit 64   # EX_USAGE
 fi
 
 file="$1"

src/cli.rs

@@ -97,7 +97,7 @@ pub enum Commands {
     /// Build specified targets (or default targets if none are given) `default`.
     Build(BuildArgs),
 
-    /// Remove build artifacts and intermediate files.
+    /// Remove build artefacts and intermediate files.
     Clean,
 
     /// Display the build dependency graph in DOT format for visualization.

tests/runner_tests.rs

@@ -119,7 +119,7 @@ fn run_executes_ninja_without_persisting_file() {
     // Ensure no ninja file remains in project directory
     assert!(!temp.path().join("build.ninja").exists());
 
-    // Drop the fake ninja artifacts. PATH is restored by guard drop.
+    // Drop the fake ninja artefacts. PATH is restored by guard drop.
     drop(ninja_path);
 }
 
@@ -150,7 +150,7 @@ fn run_build_with_emit_keeps_file() {
     assert!(emitted.contains("build "));
     assert!(!temp.path().join("build.ninja").exists());
 
-    // Drop the fake ninja artifacts. PATH is restored by guard drop.
+    // Drop the fake ninja artefacts. PATH is restored by guard drop.
     drop(ninja_path);
 }
 
@@ -179,7 +179,7 @@ fn run_build_with_emit_creates_parent_dirs() {
     assert!(emit_path.exists());
     assert!(nested_dir.exists());
 
-    // Drop the fake ninja artifacts. PATH is restored by guard drop.
+    // Drop the fake ninja artefacts. PATH is restored by guard drop.
     drop(ninja_path);
 }